PostgreSQL备份与恢复(第三章)
Posted PostgreSQLChina
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了PostgreSQL备份与恢复(第三章)相关的知识,希望对你有一定的参考价值。
作者:徐田原
pg_probackup备份工具
1.1 pg_probackup备份简介
pg_probackup是管理PG数据库集群备份和恢复的工具。用于执行周期性备份PG实例。兼容PG9.5,,96,10,11,12,13。和其他备份解决方案相比,提供下面优势,帮助部署不同备份策略和处理大量数据。
1.2 pg_probackup功能与限制
- 增量备份。页级别的增量备份,节省磁盘空间,加速备份和转储。有3种不同增量模式,根据数据流部署不同备份策略。
- 增量转储。页级别的转储,在目标目录中重用有效未更改的页面来加速转储。
- 合并。帮助部署”incrementally updated backups”策略,无须进行完备备份。
- 验证。自动数据一致性校验和按需备份校验,无须实际数据恢复。
- 校验。使用checkdb命令按需验证PG实例。
- 保留。根据保留策略管理存档和备份。可以根据恢复实际或要保留的备份数配置保留策略。还可以为特定备份制定生存时间,过期的备份可以合并或者删除。
- 并行化。在多个并行线程上进行备份、恢复、合并、删除、验证和检验。
- 压缩。以压缩状态存储备份数据以节省磁盘空间。
- 消除重复数据。通过不复制未更改的非数据文件如_vm或者_fsm来节省磁盘空间。
- 远程操作。备份位于远程系统上的PG实例或远程恢复备份。
- 从备机上进行备份。通过从备机上进行备份来避免主机上的额外负载。
- 外部目录。备份位于数据目录PGDATA之外的文件和目录,如脚本、配置文件、日志或SQL转储文件。
- 备份Catalog。以纯文本或JSON格式获取备份列表和相应的元信息。
- 归档catalog。以纯文本或JSON格式获取所有WAL时间线和相应元信息的列表。
- 部分还原。仅还原制定的数据库或从转储中排出指定数据库。
管理备份数据:pg_probackup创建一个备份目录。这个目录存储所有备份文件包括额外的元信息,以及WAL归档。可以将不同实例的备份存储在单个目录的不同子目录中。通过该工具可以进行完整备份和增量备份。
- 全量备份,包含从头恢复数据库集群需要的所有数据文件。
- 增量备份。只存储自上次以来更改的数据。允许减小备份大小并加快备份操作。支持以下增量备份模式:
- PAGE备份。扫描自上次备份以来归档的所有WAL文件。新创建的备份仅包含WAL记录到的页面。如果这些文件大小与数据库集群文件总大小相当,则加速比较小,但备份占用空间仍然较小。
- DELTA备份。读取PGDATA目录下所有数据文件,仅拷贝自上次备份以来的脏页。无须持续归档。此外,该模式还可以施加与全量备份相同的只读IO压力。
- PTRACK备份。会动态追踪页面改动。无须持续归档。每次更新数据页时,都会在PTRACK位图标记。由于一个页面只需要PTRACK位图中的一个bit,这个位图非常小。跟踪意味着会有开销,但会显著加速增量备份。
无论选择哪种备份类型,都支持以下WAL传递策略:
- 自主备份,通过复制协议传输将集群恢复到备份时的一致性状态需要的所有WAL文件。即使没有设置连续归档,需要的WAL段也会包含在备份中。
- 归档备份,依赖于连续归档。
限制:备份和转储的PG必须block_size和wal_block_size兼容且有相同主版本号;Windows下不支持ssh远程备份;ssh远程备份时,pg_probackup的版本必须相同。
1.3 pg_probackup安装与使用
wget https://github.com/postgrespro/pg_probackup/archive/2.4.15.tar.gz
tar -zxvf 2.4.15.tar.gz
cd pg_probackup-2.2.7/
make USE_PGXS=1 PG_CONFIG=/data/postgresql/bin/pg_config top_srcdir=/home/postgres/postgresql-13.3
make USE_PGXS=1 PG_CONFIG=/data/postgresql/bin/pg_config top_srcdir=/home/postgres/postgresql-13.3 install
一旦安装了pg_probackup,请完成以下设置。
1.4 pg_probackup初始化备份目录
所谓初始化:就是创建一个备份目录;用于存放备份文件和归档日志文件。
初始化备份目录,请运行以下命令:
pg_probackup init -B backup_dir
-- 示例
pg_probackup init -B /data/postgres/probackup
pg_probackup创建备份目录,其中包含以下子目录:
– wal/-wal文件目录。
– backups/-备份文件的目录。
备份目录初始化了就可以添加备份实例。
1.5 向备份目录添加新的备份Instance
pg_probackup可以在一个备份目录中存储多个数据库集群的备份。要设置所需的子目录,必须将备份实例添加到要备份的每个数据库群集的备份目录中。
要添加新的备份实例,请运行以下命令:
pg_probackup add-instance -B backup_dir -D data_dir --instance instance_name [remote_options]
-- 本地添加备份instance 示例
pg_probackup add-instance -B /data/postgres/probackup -D /data/postgres/data --instance local_6000
– data_dir是要备份的群集的数据目录。要设置和使用pg_probackup,需要对此目录的写访问权限。
– instance_name是将存储此群集的WAL和备份文件的子目录的名称。
– remote_options是可选参数,仅当data_dir位于远程系统上时,才需要指定这些参数。后续在详细讲解,参考。
pg_probackup在备份目录的 backups 和 wal 目录下创建 instance_name 子目录。backups/instance_name 目录包含 pg_probackup.conf 控制此备份实例的pg_probackup设置的配置文件。如果使用remote_选项运行此命令,则指定的参数将添加到pg_probackup.conf。
1.6 配置PostgreSQL集群
如果使用非postgres用户需要配置以下权限
For PostgreSQL 10 or higher:
$ psql -p 5432 -U postgres
BEGIN;
CREATE ROLE backup WITH LOGIN REPLICATION;
GRANT USAGE ON SCHEMA pg_catalog TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.current_setting(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_is_in_recovery() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_start_backup(text, boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_stop_backup(boolean, boolean) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_create_restore_point(text) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_switch_wal() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_last_wal_replay_lsn() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_current_snapshot() TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.txid_snapshot_xmax(txid_snapshot) TO backup;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_control_checkpoint() TO backup;
COMMIT;
1.7 配置postgresql.conf
– 确保wal_level参数高于minimal。
– 如果要在primary上配置存档,则存档模式必须设置为“on”或“always”。要在standby下执行存档,请将此参数设置为“always’
– 设置archive_命令参数,如下所示:
archive_command = 'install_dir/pg_probackup archive-push -B backup_dir --instance instance_name --wal-file-name=%f [remote_options]'
-- 示例
/opt/pg13/bin/pg_probackup archive-push -B /data/postgres/probackup --instance local_6000 --wal-file-path=%p
-- wal-file-name=%f
1.8 创建全量备份
创建备份,命令
pg_probackup backup -B backup_dir --instance instance_name -b backup_mode
示例
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b full
1.9 查看备份
pg_probackup show -B /data/postgres/probackup
pg_probackup show -B /data/postgres/probackup --instance local_6000 -i QGQMKH
pg_probackup show -B /data/postgres/probackup --instance local_6000 --archive
1.10 基于PITR备份与恢复
-- 备份
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b full
--恢复之前需要先停库;在清理PGDATA目录以及外在的表空间目录
pg_ctl stop;
rm -rf $PGDATA/*
--恢复到新的PGDATA目录
pg_probackup restore -B /data/postgres/probackup --instance local_6000 -D /data/postgres/data6000 --recovery-target-time='2021-07-11 11:05:17'
11:05:17'
-- 恢复到原目录
pg_probackup restore -B /data/postgres/probackup --instance local_6000 --recovery-target-time='2021-07-11 11:05:17'
1.11 增量备份
-- 增量备份DELTA
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b delta
-- 增量备份page
pg_probackup backup -B /data/postgres/probackup --instance local_6000 -b page
-- https://postgrespro.com/docs/postgrespro/13/app-pgprobackup
1.12 远程备份
--set up ssh
ssh-copy-id postgres@node2
--备份库 添加备份实例
pg_probackup add-instance -B /data/postgres/probackup -D /data/postgres/data --instance pg207_6000 --remote-proto=ssh --remote-host=192.168.127.126 --remote-port=22 --remote-user=postgres --remote-path=/opt/pg13/bin --ssh-options='-o ServerAliveCountMax=5 -o ServerAliveInterval=60'
--案例库 配置的参数
--备份需要用到postgres超级用户;所以在pg_hba.conf;
host all postgres 192.168.127.129/24 trust
-- postgresql.conf
wal_level = replica
archive_mode = on
archive_command = '/opt/pg13/bin/pg_probackup archive-push -B /data/postgres/probackup --instance pg207_6000 --wal-file-path=%p --wal-file-name=%f --remote-proto=ssh --remote-host=192.168.127.129 --remote-port=22 --remote-user=postgres --remote-path=/opt/pg12/bin'
-- 备份库; 执行远程备份
pg_probackup backup -B /data/postgres/probackup --instance pg207_6000 -b full --remote-proto=ssh --remote-host=192.168.127.126 --remote-port=22 --remote-user=postgres --remote-path=/opt/pg12/bin --ssh-options='-o ServerAliveCountMax=5 -o ServerAliveInterval=60'
--Take a FULL backup:
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b FULL --stream --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
--Take an incremental backup in the DELTA mode:
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b delta --stream --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
--
pg_probackup-11 set-config -B /mnt/backups --instance 'pg-11' --remote-host=postgres_host --remote-user=postgres -U backup -d backupdb
pg_probackup-11 backup -B /mnt/backups --instance 'pg-11' -b delta --stream
--pg_probackup-11 show-config -B /mnt/backups --instance 'pg-11'
--pg_probackup-11 show -B /mnt/backups --instance 'pg-11'
-- 备份库;查看备份
pg_probackup show -B /data/postgres/probackup --instance pg207_6000
1.13 远程恢复
-- 恢复
pg_probackup restore -B /data/postgres/probackup --instance local_6000 --recovery-target-time='2021-07-11 14:54:55'
1.14 delta模式和page模式的备份性能差异
测试目的:
本次测试pg_probackup工具的delta模式和page模式的备份性能差异
测试准备:
准备10亿条数据(数据大小146G):
pgbench -i -s 10000 -U backup -h 127.0.0.1 pgbench
pgbench=# \\dt
List of relations
Schema | Name | Type | Owner
--------+------------------+-------+--------
public | pgbench_accounts | table | backup
public | pgbench_branches | table | backup
public | pgbench_history | table | backup
public | pgbench_tellers | table | backup
(4 rows)
pgbench=# select count(*) from pgbench_accounts;
count
------------
1000000000
(1 row)
磁盘占用:
[postgres@ecos75]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.5G 0 7.5G 0% /dev
tmpfs 7.5G 12K 7.5G 1% /dev/shm
tmpfs 7.5G 9.2M 7.5G 1% /run
tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root 20G 9.9G 8.8G 53% /
/dev/sda1 501M 335M 130M 73% /boot
/dev/sda2 524M 12K 524M 1% /boot/efi
/dev/sdb1 296G 147G 134G 53% /postgresql/data
/dev/sdc1 296G 65M 281G 1% /postgresql/backup
tmpfs 1.5G 0 1.5G 0% /run/user/0
/dev/sdd1 296G 122G 160G 44% /postgresql/archive
全量备份(DELTA)
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b full
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b full --temp-slot
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLOIL1, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: PGDATA size: 146GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 33m:19s
2021-07-11 16:12:20.388 CST [24069] LOG: restore point 'pg_probackup, backup_id QLOIL1' created at 23/C0000178
2021-07-11 16:12:20.388 CST [24069] STATEMENT: SELECT pg_catalog.pg_create_restore_point($1)
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 15s
INFO: Validating backup QLOIL1
INFO: Backup QLOIL1 data files are valid
INFO: Backup QLOIL1 resident size: 146GB
INFO: Backup QLOIL1 completed
时间:2021-07-11 15:39 ~ 2021-07-11 16:31
ps:出现如下错误,需要配置pg lib库环境变量:
export LD_LIBRARY_PATH=/usr/local/postgresql-13.3/lib
ERROR: could not connect to database postgres: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket '/var/run/postgresql/.s.PGSQL.5432'?
增量数据准备:
插入一亿条增量数据(约15G数据)
pgbench -i -s 1000 -U probackup -h 127.0.0.1 pgbench_inc
磁盘占用:
[postgres@ecos75]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.5G 0 7.5G 0% /dev
tmpfs 7.5G 12K 7.5G 1% /dev/shm
tmpfs 7.5G 9.1M 7.5G 1% /run
tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root 20G 9.9G 8.8G 53% /
/dev/sda1 501M 335M 130M 73% /boot
/dev/sda2 524M 12K 524M 1% /boot/efi
tmpfs 1.5G 0 1.5G 0% /run/user/0
/dev/sdb1 296G 162G 119G 58% /postgresql/data
/dev/sdc1 296G 147G 134G 53% /postgresql/backup
/dev/sdd1 296G 13G 268G 5% /postgresql/archive
增量备份(DELTA)
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b delta --temp-slot
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup --stream -b delta --temp-slot
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLOMCV, backup mode: DELTA, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Parent backup: QLOIL1
INFO: PGDATA size: 161GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 31m:29s
2021-07-11 17:32:00.310 CST [30184] LOG: restore point 'pg_probackup, backup_id QLOMCV' created at 26/C1000178
2021-07-11 17:32:00.310 CST [30184] STATEMENT: SELECT pg_catalog.pg_create_restore_point($1)
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup QLOMCV
INFO: Backup QLOMCV data files are valid
INFO: Backup QLOMCV resident size: 15GB
INFO: Backup QLOMCV completed
时间:2021-07-11 17:00 ~ 2021-07-11 17:35
全量备份(PAGE)
修改postgresql.conf
vi postgresql.conf
archive_command='pg_probackup archive-push -B /postgresql/backup --instance pgbak --wal-file-name=%f'
删除增量库pgbench_inc
postgres=# drop database pgbench_inc;
pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup -b full
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -j 4 -U probackup -b full
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLILZE, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Wait for WAL segment /postgresql/backup/wal/pgbak/0000000100000020000000AE to be archived
INFO: PGDATA size: 146GB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 38m:52s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executedv
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 14s
INFO: Validating backup QLILZE
INFO: Backup QLILZE data files are valid
INFO: Backup QLILZE resident size: 146GB
INFO: Backup QLILZE completed
时间:2021-07-12 08:57 ~ 2021-07-12 09:54
增量数据准备:
插入一亿条增量数据(约15G数据)
pgbench -i -s 1000 -U probackup -h 127.0.0.1 pgbench_inc
磁盘占用:
[postgres@ecos75]$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.5G 0 7.5G 0% /dev
tmpfs 7.5G 12K 7.5G 1% /dev/shm
tmpfs 7.5G 18M 7.5G 1% /run
tmpfs 7.5G 0 7.5G 0% /sys/fs/cgroup
/dev/mapper/vg00-lv_root 20G 9.9G 8.8G 53% /
/dev/sda1 501M 335M 130M 73% /boot
/dev/sda2 524M 12K 524M 1% /boot/efi
/dev/sdb1 296G 162G 119G 58% /postgresql/data
/dev/sdc1 296G 159G 122G 57% /postgresql/backup
tmpfs 1.5G 0 1.5G 0% /run/user/0
/dev/sdd1 296G 122G 159G 44% /postgresql/archive
增量备份(PAGE)
pg_probackup-12 backup -B /postgresql/backup/ --instance pgbak -b PAGE -U probackup
[postgres@ecos75]$ pg_probackup backup -B /postgresql/backup/ --instance pgbak -b page -j 4 -U probackup
INFO: Backup start, pg_probackup version: 2.4.15, instance: pgbak, backup ID: QLPZ42, backup mode: PAGE, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without them. Reinitialize PGDATA with option '--data-checksums'.
INFO: Wait for WAL segment /postgresql/backup/wal/pgbak/0000000100000029000000C8 to be archived
INFO: Parent backup: QLPUN6
INFO: PGDATA size: 161GB
INFO: Extracting pagemap of changed blocks
INFO: Pagemap successfully extracted, time elapsed: 142 sec
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 5m:16s
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 8s
INFO: Validating backup QLPZ42
INFO: Backup QLPZ42 data files are valid
INFO: Backup QLPZ42 resident size: 15GB
INFO: Backup QLPZ42 completed
时间:2021-07-12 10:33 ~ 2021-07-12 10:44
测试结果:
本次只提供了测试流程和其中一次测试结果:
模式DELTAPAGE全量备份数据量:146G/10亿 总耗时:52min 数据传输:34min 数据验证:18min数据量:146G/10亿 总耗时:57min 数据传输:39min 数据验证:18min增量备份数据量:15G/1亿 总耗时:35min 数据传输:31min 数据验证:4min数据量:15G/1亿 总耗时:11min 数据传输:8min 数据验证:3min
通过本次测试发现,delta模式与page模式全量备份差异不明显,增量备份方面,page模式比delta模式备份效率高3倍(在这个数据量级下,数据量越大,差异或许更大)
以上是关于PostgreSQL备份与恢复(第三章)的主要内容,如果未能解决你的问题,请参考以下文章
PostgreSQL技术分享公开课:备份恢复与Point-in-Time Recovery(PITR