ES索引迁移

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ES索引迁移相关的知识，希望对你有一定的参考价值。

参考技术A 再对索引的分片数量或者是字段分词器进行调整时，需要对索引进行重建迁移，对应操作API为 reindex 。

es索引迁移Snapshot-迁移部分索引

Snapshot-迁移部分索引

源集群 192.168.40.180 192.168.40.181 192.168.40.182

目标集群 192.168.40.61 192.168.40.62 192.168.40.63

生产 需要额外，挂载硬盘

Snapshot API 是 Elasticsearch 用于对数据进行备份和恢复的一组 API 接口，可以通过 Snapshot API 进行跨集群的数据迁移，原理就是从源 Elasticsearch 集群创建数据快照，然后在目标 Elasticsearch 集群中进行恢复。

第一步：在源集群注册 Repository

创建快照前先要注册 Repository , 一个 Repository 可以包含多份快照文件， Repository 主要有以下几种类型：

fs: 共享文件系统，将快照文件存放于文件系统中
 url: 指定文件系统的URL路径，支持协议：http,https,ftp,file,jar
 s3: AWS S3对象存储,快照存放于S3中，以插件形式支持
 hdfs: 快照存放于hdfs中，以插件形式支持
 azure: 快照存放于azure对象存储中，以插件形式支持
 gcs: 快照存放于google cloud对象存储中，以插件形式支持

1模拟数据

curl -H Content-type: application/json -XPUT http://192.168.40.180:9200/_template/blog-template -d 
  "index_patterns": [
   "blog-*"
  ],
  "settings": 
   "number_of_shards": 3,
   "number_of_replicas": 1
  ,
  "mappings": 
   "dynamic_templates": [
    
     "integers": 
      "match": "int_*",
      "mapping": 
       "type": "integer"
      
     
    ,
    
     "strings": 
      "match_mapping_type": "string",
      "mapping": 
       "type": "keyword",
       "ignore_above": 256
      
     
    
   ],
   "properties": 
    "title": 
     "type": "text"
    
   
  
 
 
 
 
 
 
 
 
 
 
 
 curl -H Content-type: application/json -XPUT http://192.168.40.180:9200/blog-test2 -d 
 
 
 
 
 
 
 
 
 
 curl -H Content-type: application/x-ndjson -XPOST http://192.168.40.180:9200/blog-test/_bulk -d "index":
 "title":"elasticsearch best practice","author_firstname":"tony","author_lastname":"song","tags":"elasticsearch ,logstash","int_age":18,"locale":"zh,en"
 "index":
 "title":"elastic stack release V5","author_firstname":"shay","author_lastname":"banon","tags":"elk ,elasticsearch,release","int_age":28,"locale":"zhc,en"
 "index":
 "title":"kibana tutorial","author_firstname":"zen","author_lastname":"wei","tags":"","int_age":38,"locale":"zhc,en"
 
 
 
 
 
 
 
 curl -H Content-type: application/x-ndjson -XPOST http://192.168.40.180:9200/blog-test2/_bulk -d "index":
 "title":"elasticsearch best practice","author_firstname":"tony","author_lastname":"song","tags":"elasticsearch ,logstash","int_age":18,"locale":"zh,en"
 "index":
 "title":"elastic stack release V5","author_firstname":"shay","author_lastname":"banon","tags":"elk ,elasticsearch,release","int_age":28,"locale":"zhc,en"
 "index":
 "title":"kibana tutorial","author_firstname":"zen","author_lastname":"wei","tags":"","int_age":38,"locale":"zhc,en"

2查看索引信息

[es@master elasticsearch-7.2.0]$ curl --user elastic:1qaz1qaz -XGET http://192.168.40.180:9200/_cat/indices?pretty
 green open blog-test   lREj4IwtS9-rHuIdiTMmwA 3 1 0 0  1.4kb   743b
 green open emp         ZAEKZhqJS5i7zt-L8MkB-Q 1 1 1 0  7.5kb  3.7kb
 green open test10      ifIzAccwTc290xcfuxifJA 5 0 1 0  4.8kb  4.8kb
 green open test        4655Yhv1QPGRb-W78qVm4g 1 1 2 3   18kb    9kb
 green open blog-test2  znzIobHlQ5in6dxAr4L_eg 3 1 0 0  1.4kb   743b
 green open test4       5XnbLGmoTo-mPzIT-FpFDA 3 1 0 0  1.6kb   849b
 green open .security-7 iwX-Pv_MSRWb7I1MoP6j4Q 1 1 6 0 39.6kb 19.8kb
 green open test3       KS7T5LvPQ4WhLyihz4AJ_g 3 1 0 0  1.6kb   849b

3安装sshfs

yum install -y epel-release
 yum -y install fuse-sshfs

老集群挂载新集群目录

sshfs -o allow_other -o nonempty 192.168.40.61:/elastic/backup /elastic/backup
 （所有节点执行）除了40.61

4修改elasticsearch配置文件

修改 elasticsearch.yml 配置文件，增加如下配置:

path.repo: ["/elastic/backup"]
 
 mkdir -p /elastic/backup
 chown  es.es /elastic/backup -R
 
  chmod  777 /elastic/ -R
 chmod  777 /elastic/backup/ -R

5重启elasticsearch进程，查看创建的repo

curl --user elastic:1qaz1qaz  -H Content-type: application/json -XGET http://192.168.40.180:9200/_cluster/settings?include_defaults&filter_path=*.path.repo&pretty
 
[es@master elasticsearch-7.2.0]$ curl --user elastic:1qaz1qaz  -H Content-type: application/json -XGET http://192.168.40.180:9200/_cluster/settings?include_defaults&filter_path=*.path.repo&pretty
 
   "defaults" : 
     "path" : 
       "repo" : [
         "/elastic/backup"
       ]

6创建仓库

---会报错--忽略 -查看仓库（报错原因 需要共享文件服务"type": "fs",）我用的sshfs

curl --user elastic:1qaz1qaz  -H "Content-Type: application/json" -XPUT  http://192.168.40.182:9200/_snapshot/my_fs_backup3 -d 

    "type": "fs",
    "settings": 
        "location": "/elastic/backup",
        "max_restore_bytes_per_sec" : "50m",
        "compress": true,
        "max_snapshot_bytes_per_sec" : "50m"

7查看仓库

[root@node1 ~]# curl --user elastic:1qaz1qaz -XGET http://192.168.40.180:9200/_snapshot/_all?pretty

  "my_fs_backup" : 
    "type" : "fs",
    "settings" : 
      "location" : "/elastic/backup",
      "max_restore_bytes_per_sec" : "50m",
      "compress" : "true",
      "max_snapshot_bytes_per_sec" : "50m"
    
  



GET _snapshot/my_fs_backup
#输出结果:

  "my_fs_backup" : 
    "type" : "fs",
    "settings" : 
      "compress" : "true",
      "location" : "/home/elastic/backup/my_fs_backup"
    
  

查看在哪个node上创建了关联：
[root@master backup]# curl --user elastic:1qaz1qaz  -H Content-type: application/json -XPOST http://192.168.40.180:9200/_snapshot/my_fs_backup/_verify?pretty -d undefined


  "nodes" : 
    "I-au_rvbQRKj1xrjvWZPnA" : 
      "name" : "node2"
    ,
    "opk7TQctTPaacJ42if-dbg" : 
      "name" : "master"
    ,
    "H55oT25BS0-K-Z_WadWxSg" : 
      "name" : "node1"

8备份

emp开头索引,test开头索引,blog开头索引, 除了-.security

[root@master backup]# curl --user elastic:1qaz1qaz -H Content-type: application/json -XPUT http://192.168.40.180:9200/_snapshot/my_fs_backup/es_bak_20220917 -d "indices": "emp*,test*,blog*,-.security*,","ignore_unavailable": "true","include_global_state": false

"accepted":true[root@master backup]#

查看所有快照信息

curl --user elastic:1qaz1qaz -H Content-type: application/json -XGET http://192.168.40.180:9200/_snapshot/my_fs_backup/_all?pretty

查看指定快照信息

[root@master backup]# curl --user elastic:1qaz1qaz -H Content-type: application/json -XGET http://192.168.40.180:9200/_snapshot/my_fs_backup/es_bak*?pretty

  "snapshots" : [
    
      "snapshot" : "es_bak_20220917",
      "uuid" : "t4HY5qfZRu2wzSzXnn6NaA",
      "version_id" : 7020099,
      "version" : "7.2.0",
      "indices" : [
        "blog-test",
        "emp",
        "test10",
        "test",
        "blog-test2",
        "test4",
        "test3"
      ],
      "include_global_state" : false,
      "state" : "SUCCESS",
      "start_time" : "2022-09-16T17:41:48.692Z",
      "start_time_in_millis" : 1663350108692,
      "end_time" : "2022-09-16T17:41:49.092Z",
      "end_time_in_millis" : 1663350109092,
      "duration_in_millis" : 400,
      "failures" : [ ],
      "shards" : 
        "total" : 19,
        "failed" : 0,
        "successful" : 19
      
    
  ]

查看快照状态

curl  --user elastic:1qaz1qaz -H Content-type: application/json -XGET http://192.168.40.180:9200/_snapshot/my_fs_backup/es_bak_20220917/_status?pretty

第二步：目标集群 

path.repo: ["/elastic/backup"]

重启

查看repo

curl --user elastic:1qaz1qaz  -H Content-type: application/json -XGET http://192.168.40.61:9200/_cluster/settings?include_defaults&filter_path=*.path.repo&pretty

1新集群仓库创建为只读

curl --user elastic:1qaz1qaz  -H "Content-Type: application/json" -XPUT  http://192.168.40.61:9200/_snapshot/my_fs_backupNEW -d 

    "type": "fs",
    "settings": 
        "location": "/elastic/backup",
        "max_restore_bytes_per_sec" : "50m",
        "compress": true,
        "max_snapshot_bytes_per_sec" : "50m",
        "readonly": true

查看在那个node 上创建了关联

curl --user elastic:1qaz1qaz  -H Content-type: application/json -XPOST http://192.168.40.61:9200/_snapshot/my_fs_backupNEW/_verify?pretty -d undefined

  "nodes" : 
    "FmhibG9HTAqfd98tykZqEA" : 
      "name" : "node1"
    ,
    "SwOuMfpiRLKECWtfDx2ImA" : 
      "name" : "node2"
    ,
    "a5vPcQRSRR-f1fqnZxPIkQ" : 
      "name" : "master"

2查看快照信息

[es@master ~]$ curl --user elastic:1qaz1qaz -H Content-type: application/json -XGET http://192.168.40.61:9200/_snapshot/my_fs_backupNEW/_all?pretty


  "snapshots" : [
    
      "snapshot" : "es_bak_20220917",
      "uuid" : "t4HY5qfZRu2wzSzXnn6NaA",
      "version_id" : 7020099,
      "version" : "7.2.0",
      "indices" : [
        "blog-test",
        "emp",
        "test10",
        "test",
        "blog-test2",
        "test4",
        "test3"
      ],
      "include_global_state" : false,
      "state" : "SUCCESS",
      "start_time" : "2022-09-16T17:41:48.692Z",
      "start_time_in_millis" : 1663350108692,
      "end_time" : "2022-09-16T17:41:49.092Z",
      "end_time_in_millis" : 1663350109092,
      "duration_in_millis" : 400,
      "failures" : [ ],
      "shards" : 
        "total" : 19,
        "failed" : 0,
        "successful" : 19
      
    
  ]

[es@master ~]$

3恢复索引

curl --user elastic:1qaz1qaz -H Content-type: application/json -XPOST http://192.168.40.61:9200/_snapshot/my_fs_backupNEW/es_bak_20220917/_restore -d 
 "indices": "*",
 "ignore_unavailable": true

4查看索引 集群 节点

curl --user elastic:1qaz1qaz  "http://192.168.40.61:9200/_cat/health?v"

curl --user elastic:1qaz1qaz  "http://192.168.40.61:9200/_cat/nodes?v"
curl --user elastic:1qaz1qaz  "http://192.168.40.61:9200/_cat/indices?v"

查看恢复进度
 curl --user elastic:1qaz1qaz -H Content-type: application/json -XGET http://192.168.40.61:9200/_recovery?pretty

集群迁移部分索引成功

第三步：迁移索引模板

1导出集群中所有模板

curl  --user elastic:1qaz1qaz http://192.168.40.61:9200/_cat/templates | awk print $1| while read line ;do curl -s -u elastic:xrRWYqr2QEXZ8h\\!@6  -XGET http://192.168.40.61:9200/_template/$line?pretty >template-$line.json;done
 
导出集群中所有模板

2.模板json文件处理一遍，删除第二行和最后一行：

sed -i 2d ./*json

sed -i $d ./*json

3新集群导入【需要迁移】的索引es模板：

ls | grep json | awk -F"template-" print $2 | awk -F".json" print $1 | while read line;do curl -XPUT http://localhost:9200/_template/$line -H Content-Type: application/json -d@template-$line.json;done

第四步

sshfs进程停掉

以上是关于ES索引迁移的主要内容，如果未能解决你的问题，请参考以下文章

ES数据库重建索引——Reindex(数据迁移)

es reindex使用

ES索引和别名

es修改索引时间格式

[elasticsearch]ES迁移笔记

ElasticSearch数据迁移