rsync+inotify实现实时同步增量备份

Posted 2021-01-01

主机A：被备份的源主机
主机B：备份的目的主机

• 在linux内核中，默认的inotify机制提供了三个调控参数

[[email protected] html]# uname -r
3.10.0-693.el7.x86_64
[[email protected] html]# ll /proc/sys/fs/inotify/
total 0
-rw-r--r--. 1 root root 0 Aug 25 09:32 max_queued_events  #监控事件队列
-rw-r--r--. 1 root root 0 Aug 25 09:32 max_user_instances  #最多监控实例数
-rw-r--r--. 1 root root 0 Aug 25 09:32 max_user_watches  #每个实例最多监控文件数

[[email protected] html]# cat /proc/sys/fs/inotify/max_queued_events 
16384
[[email protected] html]# cat /proc/sys/fs/inotify/max_user_instances 
128
[[email protected] html]# cat /proc/sys/fs/inotify/max_user_watches 
8192

• 当要监控的目录、文件数量较多或者变化较频繁时，要加大这三个参数的值，如下

[[email protected] html]# vim /etc/sysctl.conf                      
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

fs.inotify.max_queued_events = 32768
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 305564824
~                                                    

[[email protected] html]# sysctl -p
fs.inotify.max_queued_events = 32768
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 305564824

[[email protected] html]# cat /proc/sys/fs/inotify/max_queued_events  
32768
[[email protected] html]# cat /proc/sys/fs/inotify/max_user_instances  
1024
[[email protected] html]# cat /proc/sys/fs/inotify/max_user_watches
305564824

• 上传inotify-tools-3.13.tar.gz并解压

[[email protected] ~]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring inotify-tools-3.13.tar.gz...
  100%     380 KB     380 KB/sec    00:00:01       0 Errors  

[[email protected] ~]# pwd
/root
[[email protected] ~]# ll | grep inotify
-rw-r--r--. 1 root root 389473 Aug 25 11:02 inotify-tools-3.13.tar.gz

[[email protected] ~]# tar -zxvf inotify-tools-3.13.tar.gz 
[[email protected] ~]# cd inotify-tools-3.13/
[[email protected] inotify-tools-3.13]# ll
total 1352
-rw-r--r--. 1 guest users 267482 Dec 30  2007 aclocal.m4
-rw-r--r--. 1 guest users     39 Dec 30  2007 AUTHORS
-rw-r--r--. 1 guest users  11301 Dec 30  2007 ChangeLog
-rwxr-xr-x. 1 guest users  45126 Dec 30  2007 config.guess
-rw-r--r--. 1 guest users   2025 Dec 30  2007 config.h.in
-rwxr-xr-x. 1 guest users  32931 Dec 30  2007 config.sub
-rwxr-xr-x. 1 guest users 690681 Dec 30  2007 configure
-rw-r--r--. 1 guest users   1829 Dec 30  2007 configure.ac
-rw-r--r--. 1 guest users  18002 Dec 30  2007 COPYING
-rwxr-xr-x. 1 guest users  15936 Dec 30  2007 depcomp
-rw-r--r--. 1 guest users   9498 Dec 30  2007 INSTALL
-rwxr-xr-x. 1 guest users   9233 Dec 30  2007 install-sh
drwxrwxrwx. 3 guest users     55 Dec 30  2007 libinotifytools
-rw-r--r--. 1 guest users 201510 Dec 30  2007 ltmain.sh
-rw-r--r--. 1 guest users    191 Dec 30  2007 Makefile.am
-rw-r--r--. 1 guest users  19389 Dec 30  2007 Makefile.in
drwxrwxrwx. 2 guest users     87 Dec 30  2007 man
-rwxr-xr-x. 1 guest users  11014 Dec 30  2007 missing
-rw-r--r--. 1 guest users     48 Dec 30  2007 NEWS
-rw-r--r--. 1 guest users    246 Dec 30  2007 README
drwxrwxrwx. 2 guest users    119 Dec 30  2007 src

• 安装

[[email protected] inotify-tools-3.13]# ./configure
[[email protected] inotify-tools-3.13]# make -j 4
[[email protected] inotify-tools-3.13]# echo $?            
0
[[email protected] inotify-tools-3.13]# make install

• ssh验证

[[email protected] ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory ‘/root/.ssh‘.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:djfNU67i/01V6z7FvTTr0sIWDaMJEXxwjiLQUHXG8dM [email protected]
The key‘s randomart image is:
+---[RSA 2048]----+
|   o+...o*+.     |
|    ..  o+=..    |
|     . . .o+ E  o|
|      . ..  .= oo|
|        S o = Bo+|
|       . . + o.*=|
|            o =.*|
|           . *.*.|
|            o.=++|
+----[SHA256]-----+
[[email protected] ~]# 
[[email protected] ~]# ssh-copy-id [email protected] 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host ‘192.168.10.178 (192.168.10.178)‘ can‘t be established.
ECDSA key fingerprint is SHA256:plDF/rfSdZkeOwQLGhV7M9NlvcQR78+4n1zWFE/Kw8E.
ECDSA key fingerprint is MD5:c1:79:b6:7e:a3:bb:52:50:d4:47:59:46:8c:87:ad:ee.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]‘s password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh ‘[email protected]‘"
and check to make sure that only the key(s) you wanted were added.

此时ssh登录主机B时候无需输入密码

• 脚本

  脚本思路：只要检测到变动或事件时候，执行rsync上行同步操作即可

在主机A上编写脚本并执行

[[email protected] ~]# vim superbackup.sh
#!/bin/bash
inotifywait -mrq -e create,move,delete,modify /var/www/html/ | while read a b c
do
        rsync -azP  --delete /var/www/html/ [email protected]:/web-back

done
~                                                   

[[email protected] ~]# ./superbackup.sh

• 测试，在主机A上被备份的目录/var/www/html下修改文件

[[email protected] html]# vim superbackup.txt 
如果你看到这个文件，说明实时同步成功
!!!are you ok
~                                                   
[[email protected] html]# echo "再次新增文件，如果看到这个文件，说明OK" > superbackup1.txt

验证，在主机B查看目的备份目录/web-back/

[[email protected] ~]# ll /web-back/
总用量 24
-rw-r--r--. 1 root root 31 8月  24 23:35 1.txt
-rw-r--r--. 1 root root 31 8月  24 23:43 2.txt
-rw-r--r--. 1 root root 21 8月  24 23:09 ajie.txt
-rw-r--r--. 1 root root 14 8月  24 23:15 index.txt
-rw-r--r--. 1 root root 69 8月  25 13:24 superbackup.txt
-rw-r--r--. 1 root root 49 8月  24 22:36 testbackup.txt
[[email protected] ~]# 
[[email protected] ~]# ll /web-back/     
总用量 28
-rw-r--r--. 1 root root 31 8月  24 23:35 1.txt
-rw-r--r--. 1 root root 31 8月  24 23:43 2.txt
-rw-r--r--. 1 root root 21 8月  24 23:09 ajie.txt
-rw-r--r--. 1 root root 14 8月  24 23:15 index.txt
-rw-r--r--. 1 root root 57 8月  25 13:27 superbackup1.txt
-rw-r--r--. 1 root root 69 8月  25 13:24 superbackup.txt
-rw-r--r--. 1 root root 49 8月  24 22:36 testbackup.txt

[[email protected] ~]# cat /web-back/superbackup.txt  
如果你看到这个文件，说明实时同步成功
!!!are you ok
[[email protected] ~]# cat /web-back/superbackup1.txt 
再次新增文件，如果看到这个文件，说明OK
[[email protected] ~]# 

• 优化脚本

将主机A下的/backup/super/V2目录备份到主机B的/web-back目录下

脚本如下

点我下载

在主机下的/backup/super/V2目录先创建文件供测试用

[[email protected] ~]# mkdir -p /backup/super/V2
[[email protected] ~]# touch /backup/super/V2/index.php
[[email protected] ~]# touch /backup/super/V2/index.txt
[[email protected] ~]# touch /backup/super/V2/index.html

• 让脚本开机后自动后台执行

[[email protected] ~]# echo ‘/root/superbackupV2.sh &‘ >> /etc/rc.local 

[[email protected] ~]# cat !$ 
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run ‘chmod +x /etc/rc.d/rc.local‘ to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
/root/superbackupV2.sh &

[[email protected] ~]# chmod +x /etc/rc.d/rc.local

验证

主机A重启后修改/backup/super/V2/index.php

[[email protected] ~]# echo "这是重启后修改，如果你看到这个文件，说明superbackupV2执行成功" > /backup/super/V2/index.php 

在主机B查看备份目录

[[email protected] ~]# ll /web-back/V2/
总用量 4
-rw-r--r--. 1 root root  0 8月  25 14:07 index.html
-rw-r--r--. 1 root root 86 8月  25 14:42 index.php
-rw-r--r--. 1 root root  0 8月  25 14:07 index.txt
[[email protected] ~]# cat /web-back/V2/index.php
这是重启后修改，如果你看到这个文件，说明superbackupV2执行成功
[[email protected] ~]# 

[[email protected] ~]# ps aux | grep inotify
root        976  0.0  0.0   6524   628 ?        S    14:49   0:00 inotifywait -mrq -e create,move,delete,modify,attrib /backup/super/V2

• rsycn常用参数

-a, --archive 归档模式，表示以递归方式传输文件，并保持所有文件属性，等于-rlptgoD
-p, --perms 保持文件权限
-P 等同于 --partial 保留那些因故没有完全传输的文件，以是加快随后的再次传输
-q, --quiet 精简输出模式
-r, --recursive 对子目录以递归模式处理
-t, --times 保持文件时间信息
-v, --verbose 详细模式输出
-z, --compress 对备份的文件在传输时进行压缩处理
--delete 删除那些DST中SRC没有的文件

• inotifywait常用参数

-m, 持续监视变化
-r, 使用递归形式监视目录
-q, 减少冗余信息，只打印出需要的信息
-e, 指定要监视的事件列表

可监视的事件
事件  描述
access  访问，读取文件。
modify  修改，文件内容被修改。
attrib  属性，文件元数据被修改。
move    移动，对文件进行移动操作。
create  创建，生成新文件
open    打开，对文件进行打开操作。
close   关闭，对文件进行关闭操作。
delete  删除，文件被删除。