ElasticSearch索引快照

Posted 2020-10-10 浪迹天涯

参考文档：https://www.elastic.co/guide/en/elasticsearch/reference/5.1/modules-snapshots.html

备份数据之前，要创建一个仓库来保存数据，仓库的类型支持Shared filesystem, Amazon S3, HDFS和Azure Cloud。
fs仓库示例：
需要在elasticsearch.yml添加如下配置：
path.repo: ["/data1/esdata/backup"]
注意：这里fs是Shared filesystem, such as a NAS，本地文件集群不支持，单节点可以用本地文件
curl -XPUT http://127.0.0.1:9200/_snapshot/backup_feeds -d‘
{
"type": "fs",
"settings": {
"location": "feeds"
}
}‘

hdfs仓库示例：
a.每个节点安装插件
/opt/17173/elasticsearch/bin/elasticsearch-plugin install repository-hdfs
b.从hadoop复制配置文件hdfs-site.xml和core-site.xml到elasticsearch/config目录下
c./opt/17173/elasticsearch/plugins/repository-hdfs/plugin-security.policy增加如下配置,注意行注释不要用#
permission java.lang.RuntimePermission "accessDeclaredMembers";
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "shutdownHooks";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission javax.security.auth.AuthPermission "doAs";
permission javax.security.auth.AuthPermission "getSubject";
permission javax.security.auth.AuthPermission "modifyPrivateCredentials";
permission java.security.AllPermission;
permission java.util.PropertyPermission "*", "read,write";
permission javax.security.auth.PrivateCredentialPermission "org.apache.hadoop.security.Credentials * \"*\"", "read";
d.config/jvm.options增加一行
-Djava.security.policy=file:///opt/17173/elasticsearch/plugins/repository-hdfs/plugin-security.policy

e.以上配置每个节点保持一致，然后逐个节点重启

1.创建一个仓库（creating the repository）
curl -XPUT http://127.0.0.1:9200/_snapshot/backup_feeds -d‘
{
"type": "hdfs",
"settings": {
"uri": "hdfs://hadoop73:9000/",
"path": "elasticsearch/respositories/feeds",
"compress": "true",
"chunk_size": "10mb",
"load_defaults": "true",
"user":"hadoop",
"conf_location": ["/opt/17173/elasticsearch/core-site.xml","/opt/17173/elasticsearch/hdfs-site.xml"]
}
}‘

查看仓库信息
curl -XGET ‘http://localhost:9200/_snapshot/backup_feeds?pretty‘

2.备份索引
curl -XPUT http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_20170920?wait_for_completion=true?pretty -d‘
{
"indices": "feed_info,play_following"
}‘

所有索引进行快照：
 curl -XPUT "localhost:9200/_snapshot/backup_feeds/snapshot_20170920?wait_for_completion=true?pretty"

3、查看快照信息
curl -XGET http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_20170920
查看所有快照信息
curl -XGET http://127.0.0.1:9200/_snapshot/backup_feeds/_all
查看快照状态
curl -XGET http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_20170920/_status
4.删除备份
curl -XDELETE http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_20170920

5恢复
curl -XPOST http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_20170920/_restore
　　 恢复所有索引：
curl -XPOST "localhost:9200/_snapshot/backup_feeds/snapshot_20170920/_restore?wait_for_completion=true"
　　　恢复某个索引：
curl -XPOST "localhost:9200/_snapshot/backup_feeds/snapshot_20170920/_restore?wait_for_completion=true" -d ‘{"indices":"feed_info","ignore_unavailable":"true"}‘
注意：
快照恢复前需要关掉索引
curl -XPOST "localhost:9200/feed_info/_close"
curl -XPOST "localhost:9200/play_following/_close"

开启索引
curl -XPOST "localhost:9200/my_index/_open"

每天定时备份索引，删除30天之前备份数据 es_backup.sh
#!/bin/bash
##每天定时备份索引，删除30天之前备份数据
#0 * * * * sh /opt/17173/elasticsearch/bin/es_backup.sh >> /tmp/es_backup.log
cd `dirname $0`
BIN_DIR=`pwd`

DATE=`date +%Y%m%d`
echo "备份ES索引：$DATE "
curl -XPUT http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_${DATE}?wait_for_completion=true?pretty -d‘
{
"indices": "feed_info,play_following"
}‘
echo ""
DEL_DATE=`date +%Y%m%d --date "30 days ago"`
echo "删除ES索引备份：$DEL_DATE "
curl -XDELETE http://127.0.0.1:9200/_snapshot/backup_feeds/snapshot_${DEL_DATE}
echo ""