Linux之rsync数据同步服务
Posted 潇潇、寒
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Linux之rsync数据同步服务相关的知识，希望对你有一定的参考价值。
rsync是一个开源、快速的、多动能的、可以实现全量，增量的本地或远程数据同步备份工具，它适用于多种操作系统平台。
1、rsync的特性（功能）
（1）支持拷贝特殊文件（如链接文件、设备文件）
（2）拷贝时可以排除目录中目录或文件不需要同步的功能
（3）可以保持源文件或目录的属性不发生改变
（4）可以实现全量、增量的同步功能、传输效率比较高
（5）可以使用scp、ssh、rsh等方式配合使用
（6）可以通过socke传输方式来同步数据
（7）支持匿名或认证（无需系统用户）的进和模式来进行传输同步数据
2、rsync的工作方式
（1）主机本地的传输，类似于cp功能
（2）借助于SSH服务来传输数据
（3）以守护进程的方式来进行传输数据
[root@localhost1 ~]# rsync --help rsync version 3.0.6 protocol version 30 Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others. Web site: http://rsync.samba.org/ Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence for details. rsync is a file transfer program capable of efficient remote update via a fast differencing algorithm. Usage: rsync [OPTION]... SRC [SRC]... DEST or rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST or rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST or rsync [OPTION]... [USER@]HOST:SRC [DEST] or rsync [OPTION]... [USER@]HOST::SRC [DEST] or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST] The \':\' usages connect via remote shell, while \'::\' & \'rsync://\' usages connect to an rsync daemon, and require SRC or DEST to start with a module name. Options -v, --verbose increase verbosity #增加冗长（显示详细模式） -q, --quiet suppress non-error messages --no-motd suppress daemon-mode MOTD (see manpage caveat) -c, --checksum skip based on checksum, not mod-time & size -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) #归档模式， --no-OPTION turn off an implied OPTION (e.g. --no-D) #关闭一个隐含的选项 -r, --recursive recurse into directories -R, --relative use relative path names --no-implied-dirs don\'t send implied dirs with --relative -b, --backup make backups (see --suffix & --backup-dir) --backup-dir=DIR make backups into hierarchy based in DIR --suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir) -u, --update skip files that are newer on the receiver --inplace update destination files in-place (SEE MAN PAGE) --append append data onto shorter files --append-verify like --append, but with old data in file checksum -d, --dirs transfer directories without recursing -l, --links copy symlinks as symlinks -L, --copy-links transform symlink into referent file/dir --copy-unsafe-links only "unsafe" symlinks are transformed --safe-links ignore symlinks that point outside the source tree -k, --copy-dirlinks transform symlink to a dir into referent dir -K, --keep-dirlinks treat symlinked dir on receiver as dir -H, --hard-links preserve hard links -p, --perms preserve permissions -E, --executability preserve the file\'s executability --chmod=CHMOD affect file and/or directory permissions -A, --acls preserve ACLs (implies --perms) -X, --xattrs preserve extended attributes -o, --owner preserve owner (super-user only) -g, --group preserve group --devices preserve device files (super-user only) --copy-devices copy device contents as regular file --specials preserve special files -D same as --devices --specials -t, --times preserve modification times -O, --omit-dir-times omit directories from --times --super receiver attempts super-user activities --fake-super store/recover privileged attrs using xattrs -S, --sparse handle sparse files efficiently -n, --dry-run perform a trial run with no changes made -W, --whole-file copy files whole (without delta-xfer algorithm) -x, --one-file-system don\'t cross filesystem boundaries -B, --block-size=SIZE force a fixed checksum block-size -e, --rsh=COMMAND specify the remote shell to use #指定要使用的远程shell --rsync-path=PROGRAM specify the rsync to run on the remote machine #在指定的远程机器上运行rsync --existing skip creating new files on receiver #跳过在接受器上创建新文件 --ignore-existing skip updating files that already exist on receiver #跳过已经存在于接收器上的更新文件 --remove-source-files sender removes synchronized files (non-dirs) #发送方删除同步文件（非目录） --del an alias for --delete-during #别名 --delete delete extraneous files from destination dirs --delete-before receiver deletes before transfer, not during --delete-during receiver deletes during transfer (default) --delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not during --delete-excluded also delete excluded files from destination dirs --ignore-errors delete even if there are I/O errors --force force deletion of directories even if not empty --max-delete=NUM don\'t delete more than NUM files --max-size=SIZE don\'t transfer any file larger than SIZE --min-size=SIZE don\'t transfer any file smaller than SIZE --partial keep partially transferred files #保持部分转移文件 --partial-dir=DIR put a partially transferred file into DIR --delay-updates put all updated files into place at transfer\'s end -m, --prune-empty-dirs prune empty directory chains from the file-list --numeric-ids don\'t map uid/gid values by user/group name --timeout=SECONDS set I/O timeout in seconds --contimeout=SECONDS set daemon connection timeout in seconds -I, --ignore-times don\'t skip files that match in size and mod-time --size-only skip files that match in size --modify-window=NUM compare mod-times with reduced accuracy -T, --temp-dir=DIR create temporary files in directory DIR -y, --fuzzy find similar file for basis if no dest file --compare-dest=DIR also compare destination files relative to DIR --copy-dest=DIR ... and include copies of unchanged files --link-dest=DIR hardlink to files in DIR when unchanged -z, --compress compress file data during the transfer #在传输过程中压缩文件数据 --compress-level=NUM explicitly set compression level #显示的设置压缩级别 --skip-compress=LIST skip compressing files with a suffix in LIST -C, --cvs-exclude auto-ignore files the same way CVS does -f, --filter=RULE add a file-filtering RULE -F same as --filter=\'dir-merge /.rsync-filter\' repeated: --filter=\'- .rsync-filter\' --exclude=PATTERN exclude files matching PATTERN --exclude-from=FILE read exclude patterns from FILE --include=PATTERN don\'t exclude files matching PATTERN --include-from=FILE read include patterns from FILE --files-from=FILE read list of source-file names from FILE -0, --from0 all *-from/filter files are delimited by 0s -s, --protect-args no space-splitting; only wildcard special-chars --address=ADDRESS bind address for outgoing socket to daemon --port=PORT specify double-colon alternate port number --sockopts=OPTIONS specify custom TCP options --blocking-io use blocking I/O for the remote shell --stats give some file-transfer stats -8, --8-bit-output leave high-bit chars unescaped in output -h, --human-readable output numbers in a human-readable format --progress show progress during transfer #在转移过程中显示进行情况 -P same as --partial --progress -i, --itemize-changes output a change-summary for all updates --out-format=FORMAT output updates using the specified FORMAT --log-file=FILE log what we\'re doing to the specified FILE --log-file-format=FMT log updates using the specified FMT --password-file=FILE read daemon-access password from FILE --list-only list the files instead of copying them --bwlimit=KBPS limit I/O bandwidth; KBytes per second --write-batch=FILE write a batched update to FILE --only-write-batch=FILE like --write-batch but w/o updating destination --read-batch=FILE read a batched update from FILE --protocol=NUM force an older protocol version to be used --iconv=CONVERT_SPEC request charset conversion of filenames -4, --ipv4 prefer IPv4 -6, --ipv6 prefer IPv6 --version print version number (-h) --help show this help (-h works with no other options) Use "rsync --daemon --help" to see the daemon-mode command-line options. Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation. See http://rsync.samba.org/ for updates, bug reports, and answers
3、rsync同步命令格式
（1）本地主机
Local: rsync [OPTION...] SRC... [DEST]
（2）借助SSH服务
Access via remote shell:
Pull: rsync [OPTION...] [USER@]HOST:SRC... [DEST]
Push: rsync [OPTION...] SRC... [USER@]HOST:DEST
[root@localhost1 ~]# ll total 20 -rw-r--r-- 1 root root 0 Feb 17 20:06 123.txt -rw-------. 1 root root 929 Feb 9 10:03 anaconda-ks.cfg -rw-r--r--. 1 root root 9615 Feb 9 10:03 install.log -rw-r--r--. 1 root root 3161 Feb 9 10:01 install.log.syslog [root@localhost1 ~]# rsync -avzP -e "ssh -p 22" root@192.168.181.129:/tmp/666 ~ root@192.168.181.129\'s password: receiving incremental file list 666 0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1) sent 30 bytes received 69 bytes 28.29 bytes/sec total size is 0 speedup is 0.00 [root@localhost1 ~]# ll total 20 -rw-r--r-- 1 root root 0 Feb 17 20:06 123.txt -rw-r--r-- 1 root root 0 Feb 17 20:06 666 -rw-------. 1 root root 929 Feb 9 10:03 anaconda-ks.cfg -rw-r--r--. 1 root root 9615 Feb 9 10:03 install.log -rw-r--r--. 1 root root 3161 Feb 9 10:01 install.log.syslog
[root@localhost2 ~]# ll total 20 -rw-------. 1 root root 929 Feb 15 02:25 anaconda-ks.cfg -rw-r--r--. 1 root root 9615 Feb 15 02:25 install.log -rw-r--r--. 1 root root 3161 Feb 15 02:23 install.log.syslog [root@localhost1 tmp]# ls 123.txt 666 caicai fenfa@192.168.181.129 yum.log [root@localhost1 tmp]# rsync -azvP -e "ssh -p 22" /tmp/666 root@192.168.181.129:~ root@192.168.181.129\'s password: sending incremental file list 666 0 100% 0.00kB/s 0:00:00 (xfer#1, to-check=0/1) sent 64 bytes received 31 bytes 27.14 bytes/sec total size is 0 speedup is 0.00 [root@localhost2 ~]# ll total 20 -rw-r--r--. 1 root root 0 Feb 17 18:55 666 -rw-------. 1 root root 929 Feb 15 02:25 anaconda-ks.cfg -rw-r--r--. 1 root root 9615 Feb 15 02:25 install.log -rw-r--r--. 1 root root 3161 Feb 15 02:23 install.log.syslog
（3）守护进程模式
Access via rsync daemon: Pull: rsync [OPTION...] [USER@]HOST::SRC... [DEST] rsync [OPTION...] rsync://[USER@]HOST[:PORT]/SRC... [DEST]
rsync -avzP rsync://root@192.168.181.129:22/tmp/666 /tmp
Push: rsync [OPTION...] SRC... [USER@]HOST::DEST rsync [OPTION...] SRC... rsync://[USER@]HOST[:PORT]/DEST
rsync -avzP /tmp/666 rsync://root@192.168.181.129:22/tmp
守护进程模式传输数据实战：
a、整体环境
备份数据的客户端
[root@localhost1 ~]# cat /etc/redhat-release CentOS release 6.8 (Final) [root@localhost1 ~]# uname -r 2.6.32-642.el6.x86_64
备份数据服务器端
[root@localhost2 ~]# cat /etc/redhat-release CentOS release 6.8 (Final) [root@localhost2 ~]# uname -r 2.6.32-642.el6.x86_64
b、检查是否安装rsync服务
[root@localhost2 ~]# rpm -qa | grep rsync rsync-3.0.6-12.el6.x86_64
[root@localhost1 ~]# rpm -qa | grep "rsync" rsync-3.0.6-12.el6.x86_64
c、配置rsync的配置文件/etc/rsyncd.conf
rsyncd.conf(5) 28 Jan 2018 NAME rsyncd.conf - configuration file for rsync in daemon mode #在守护进程模式下的配置 SYNOPSIS rsyncd.conf DESCRIPTION The rsyncd.conf file is the runtime configuration file for rsync when run as an rsync daemon. #rsyncd.conf文件是rsync的运行时配置文件，作为rsync守护进程运行 The rsyncd.conf file controls authentication, access, logging and available modules. #rsyncd.conf文件控制身份验证、访问、日志记录和可用模块。 FILE FORMAT The file consists of modules and parameters. A module begins with the name of the module in square brackets and continues until the next module begins. Modules contain parameters of the form "name = value". The file is line-based -- that is, each newline-terminated line represents either a comment, a module name or a parameter. Only the first equals sign in a parameter is significant. Whitespace before or after the first equals sign is discarded. Leading, trailing and internal whitespace in module and parameter names is irrelevant. Leading and trailing whitespace in a parameter value is discarded. Internal whitespace within a parameter value is retained verbatim. Any line beginning with a hash (#) is ignored, as are lines containing only whitespace. (If a hash occurs after anything other than leading whitespace, it is considered a part of the line\'s content.) Any line ending in a \\ is "continued" on the next line in the customary UNIX fashion. The values following the equals sign in parameters are all either a string (no quotes needed) or a boolean, which may be given as yes/no, 0/1 or true/false. Case is not significant in boolean values, but is preserved in string values. LAUNCHING THE RSYNC DAEMON #启动rsync守护进程 The rsync daemon is launched by specifying the --daemon option to rsync. #加 --daemon选项来启动 The daemon must run with root privileges if you wish to use chroot, to bind to a port numbered under 1024 (as is the default 873), or to set file ownership. Otherwise, it must just have permission to read and write the appropriate data, log, and lock files. You can launch it either via inetd, as a stand-alone daemon, or from an rsync client via a remote shell. If run as a stand-alone daemon then just run the command "rsync --daemon" from a suitable startup script. When run via inetd you should add a line like this to /etc/services: rsync 873/tcp and a single line something like this to /etc/inetd.conf: rsync stream tcp nowait root /usr/bin/rsync rsyncd --daemon Replace "/usr/bin/rsync" with the path to where you have rsync installed on your system. You will then need to send inetd a HUP signal to tell it to reread its config file. Note that you should not send the rsync daemon a HUP signal to force it to reread the rsyncd.conf file. The file is re-read on each client connection. GLOBAL PARAMETERS The first parameters in the file (before a [module] header) are the global parameters. Rsync also allows for the use of a "[global]" module name to indicate the start of one or more global-parameter sections (the name must be lower case). You may also include any module parameters in the global part of the config file in which case the supplied value will override the default for that parameter. You may use references to environment variables in the values of parameters. String parameters will have %VAR% references expanded as late as possible (when the string is used in the program), allowing for the use of variables that rsync sets at connection time, such as RSYNC_USER_NAME. Non-string parameters (such as true/false settings) are expanded when read from the config file. If a variable does not exist in the environment, or if a sequence of characters is not a valid reference (such as an un-paired percent sign), the raw characters are passed through unchanged. This helps with backward compatibility and safety (e.g. expanding a non-existent %VAR% to an empty string in a path could result in a very unsafe path). The safest way to insert a literal % into a value is to use %%. motd file #当天日志 This parameter allows you to specify a "message of the day" to display to clients on each connect. This usually contains site information and any legal notices. The default is no motd file. This can be overridden by the --dparam=motdfile=FILE command-line option when starting the daemon. pid file
这个参数告诉rsync守护进程将进程ID写入该文件。如果文件已经存在，rsync守护进程将中止而不是覆盖文件。当启动守护进程时，这可以被—dparam=pidfile=文件命令行选项覆盖
This parameter tells the rsync daemon to write its process ID to that file. If the file already exists, the rsync daemon will abort rather than overwrite the file. This can be overridden by the --dparam=pidfile=FILE command-line option when starting the daemon. port
您可以通过指定这个值(默认值为873)来覆盖守护进程监听的默认端口。如果这个守护进程是由inetd运行的，并且被——port命令行选项所取代，那么就会忽略它。
You can override the default port the daemon will listen on by specifying this value (defaults to 873). This is ignored if the daemon is being run by inetd, and is superseded by the --port command-line option. address
您可以通过指定这个值来覆盖守护进程监听的默认IP地址。如果这个守护进程是由inetd运行的，并且被——地址命令行选项所取代，那么这将被忽略。
You can override the default IP address the daemon will listen on by specifying this value. This is ignored if the daemon is being run by inetd, and is superseded by the --address command-line option. socket options This parameter can provide endless fun for people who like to tune their systems to the utmost degree. You can set all sorts of socket options which may make transfers faster (or slower!). Read the man page for the setsockopt() system call for details on some of the options you may be able to set. By default no special socket options are set. These settings can also be specified via the --sockopts command-line option. listen backlog You can override the default backlog value when the daemon listens for connections. It defaults to 5. MODULE PARAMETERS #模块参数 After the global parameters you should define a number of modules, each module exports a directory tree as a symbolic name. Modules are exported by specifying a module name in square brackets [module] followed by the parameters for that module. The module name cannot contain a slash or a closing square bracket. If the name contains whitespace, each internal sequence of whitespace will be changed into a single space, while leading or trailing whitespace will be discarded. Also, the name cannot be "global" as that exact name indicates that global parameters follow (see above). As with GLOBAL PARAMETERS, you may use references to environment variables in the values of parameters. See the GLOBAL PARAMETERS section for more details. comment This parameter specifies a description string that is displayed next to the module name when clients obtain a list of available modules. The default is no comment. path This parameter specifies the directory in the daemon\'s filesystem to make available in this module. You must specify this parameter for each module in rsyncd.conf. You may base the path\'s value off of an environment variable by surrounding the variable name with percent signs. You can even reference a variable that is set by rsync when the user connects. For example, this would use the authorizing user\'s name in the path: path = /home/%RSYNC_USER_NAME% It is fine if the path includes internal spaces -- they will be retained verbatim (which means that you shouldn\'t try to escape them). If your final directory has a trailing space (and this is somehow not something you wish to fix), append a trailing slash to the path to avoid losing the trailing whitespace. use chroot If "use chroot" is true, the rsync daemon will chroot to the "path" before starting the file transfer with the client. This has the advantage of extra protection against possible implementation security holes, but it has the disadvantages of requiring super-user privileges, of not being able to follow symbolic links that are either absolute or outside of the new root path, and of complicating the preservation of users and groups by name (see below). As an additional safety feature, you can specify a dot-dir in the module\'s "path" to indicate the point where the chroot should occur. This allows rsync to run in a chroot with a non-"/" path for the top of the transfer hierarchy. Doing this guards against unintended library loading (since those absolute paths will not be inside the transfer hierarchy unless you have used an unwise pathname), and lets you setup libraries for the chroot that are outside of the transfer. For example, specifying "/var/rsync/./module1" will chroot to the "/var/rsync" directory and set the inside-chroot path to "/module1". If you had omitted the dot-dir, the chroot would have used the whole path, and the inside-chroot path would have been "/". When both "use chroot" and "daemon chroot" are false, OR the inside-chroot path of "use chroot" is not "/", rsync will: (1) munge symlinks by default for security reasons (see "munge symlinks" for a way to turn this off, but only if you trust your users), (2) substitute leading slashes in absolute paths with the module\'s path (so that options such as --backup-dir, --compare-dest, etc. interpret an absolute path as rooted in the module\'s "path" dir), and (3) trim ".." path elements from args if rsync believes they would escape the module hierarchy. The default for "use chroot" is true, and is the safer choice (especially if the module is not read-only). When this parameter is enabled, the "numeric-ids" option will also default to being enabled (disabling name lookups). See below for what a chroot needs in order for name lookups to succeed. If you copy library resources into the module\'s chroot area, you should protect them through your OS\'s normal user/group or ACL settings (to prevent the rsync module\'s user from being able to change them), and then hide them from the user\'s view via "exclude" (see how in the discussion of that parameter). At that point it will be safe to enable the mapping of users and groups by name using the "numeric ids" daemon parameter (see below). Note also that you are free to setup custom user/group information in the chroot area that is different from your normal system. For example, you could abbreviate the list of users and groups. daemon chroot This parameter specifies a path to which the daemon will chroot before beginning communication with clients. Module paths (and any "use chroot" settings) will then be related to this one. This lets you choose if you want the whole daemon to be chrooted (with this setting), just the transfers to be chrooted (with "use chroot"), or both. Keep in mind that the "daemon chroot" area may need various OS/lib/etc files installed to allow the daemon to function. By default the daemon runs without any chrooting. numeric ids Enabling this parameter disables the mapping of users and groups by name for the current daemon module. This prevents the daemon from trying to load any user/group-related files or libraries. This enabling makes the transfer behave as if the client had passed the --numeric-ids command-line option. By default, this parameter is enabled for chroot modules and disabled for non-chroot modules. Also keep in mind that uid/gid preservation requires the module to be running as root (see "uid") or for "fake super" to be configured. A chroot-enabled module should not have this parameter enabled unless you\'ve taken steps to ensure that the module has the necessary resources it needs to translate names, and that it is not possible for a user to change those resources. That includes being the code being able to call functions like getpwuid() , getgrgid() , getpwname() , and getgrnam() . You should test what libraries and config files are required for your OS and get those setup before starting to test name mapping in rsync. munge symlinks This parameter tells rsync to modify all symlinks in the same way as the (non-daemon-affecting) --munge-links command-line option (using a method described below). This should help protect your files from user trickery when your daemon module is writable. The default is disabled when "use chroot" is on with an inside-chroot path of "/", OR if "daemon chroot" is on, otherwise it is enabled. If you disable this parameter on a daemon that is not read-only, there are tricks that a user can play with uploaded symlinks to access daemon-excluded items (if your module has any), and, if "use chroot" is off, rsync can even be tricked into showing or changing data that is outside the module\'s path (as access-permissions allow). The way rsync disables the use of symlinks is to prefix each one with the string "/rsyncd-munged/". This prevents the links from being used as long as that directory does not exist. When this parameter is enabled, rsync will refuse to run if that path is a directory or a symlink to a directory. When using the "munge symlinks" parameter in a chroot area that has an inside-chroot path of "/", you should add "/rsyncd-munged/" to the exclude setting for the module so that a user can\'t try to create it. Note: rsync makes no attempt to verify that any pre-existing symlinks in the module\'s hierarchy are as safe as you want them to be (unless, of course, it just copied in the whole hierarchy). If you setup an rsync daemon on a new area or locally add symlinks, you can manually protect your symlinks from being abused by prefixing "/rsyncd-munged/" to the start of every symlink\'s value. There is a perl script in the support directory of the source code named "munge-symlinks" that can be used to add or remove this prefix from your symlinks. When this parameter is disabled on a writable module and Linux之rsync数据同步服务Linux三阶段之四：实时同步（inotify+rsync,sersync+rsync）
linux---集群架构初探搭建backup服务器之rsync