使用elasticdump迁移数据到新es集群

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了使用elasticdump迁移数据到新es集群相关的知识，希望对你有一定的参考价值。

参考文档

https://blog.csdn.net/u012731379/article/details/79856113 
https://blog.csdn.net/u010466329/article/details/79209236 
https://blog.csdn.net/laoyang360/article/details/65449407

迁移方法

  • 通过logstash的input和output配置迁移（配置灵活适用于长期数据同步等）
  • 通过迁移工具如elasticdump等（适用于备份一次性小量数据操作）
  • 通过elasticsarch自带快照功能（适用于一次性迁移大量数据）

操作步骤

1、安装elasticdump

[[email protected]_8_24_centos ~]# yum install nodejs npm  
[[email protected]_8_24_centos ~]# npm install elasticdump 
/root
└─┬ [email protected] 
  ├── [email protected] 
  ├─┬ [email protected] 
  │ ├─┬ [email protected] 
  │ │ ├── [email protected] 
  │ │ └── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├─┬ [email protected] 
  │ │ └── [email protected] 
  │ ├── [email protected] 
  │ └─┬ [email protected] 
  │   └── [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
  │ └── [email protected] 
  ├── [email protected] 
  ├── [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
  │ └── [email protected] 
  ├─┬ [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├─┬ [email protected] 
  │ │ └── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├─┬ [email protected] 
  │ │ └── [email protected] 
  │ ├─┬ [email protected] 
  │ │ ├─┬ [email protected] 
  │ │ │ ├── [email protected] 
  │ │ │ ├── [email protected] 
  │ │ │ ├── [email protected] 
  │ │ │ └─┬ [email protected] 
  │ │ │   └── [email protected] 
  │ │ └── [email protected] 
  │ ├─┬ [email protected] 
  │ │ ├── assert-p[email protected] 
  │ │ ├─┬ [email protected] 
  │ │ │ ├── [email protected] 
  │ │ │ ├── [email protected] 
  │ │ │ └── [email protected] 
  │ │ └─┬ [email protected] 
  │ │   ├── [email protected] 
  │ │   ├── [email protected] 
  │ │   ├── [email protected] 
  │ │   ├── [email protected] 
  │ │   ├── [email protected] 
  │ │   ├── [email protected] 
  │ │   ├── [email protected] 
  │ │   └── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├─┬ [email protected] 
  │ │ └── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├── [email protected] 
  │ ├─┬ [email protected] 
  │ │ ├── [email protected] 
  │ │ └── [email protected] 
  │ └── [email protected] 
  ├─┬ [email protected] 
  │ └── [email protected] 
  └─┬ [email protected] 
    ├── [email protected] 
    └─┬ [email protected] 
      ├── [email protected] 
      ├── [email protected] 
      ├── [email protected] 
      ├── [email protected] 
      └── [email protected] 

npm WARN enoent ENOENT: no such file or directory, open ‘/root/package.json‘
npm WARN root No description
npm WARN root No repository field.
npm WARN root No README data
npm WARN root No license field.
[[email protected]_8_24_centos ~]# cd node_modules/elasticdump/bin

2、导出mapping

[[email protected]_8_24_centos bin]# ./elasticdump --input=http://10.2.3.159:9200/reconciliation  --output=http://10.2.100.24:9200/reconciliationonline --type=mapping
Mon, 11 Feb 2019 09:28:10 GMT | starting dump
Mon, 11 Feb 2019 09:28:10 GMT | got 1 objects from source elasticsearch (offset: 0)
Mon, 11 Feb 2019 09:28:16 GMT | sent 1 objects to destination elasticsearch, wrote 1
Mon, 11 Feb 2019 09:28:16 GMT | got 0 objects from source elasticsearch (offset: 1)
Mon, 11 Feb 2019 09:28:16 GMT | Total Writes: 1
Mon, 11 Feb 2019 09:28:16 GMT | dump complete
[[email protected]_8_24_centos bin]# curl  10.2.100.24:9200/_cat/indices|grep reconciliation
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 43 32578   43 14037    0     0   5694      0  0:00:05  0:00:02  0:00:03  5694yellow open reconciliationonline   KnZrsU-7TmWWBs-1-dBRrw 5 1       0  0    1.1kb    1.1kb

3、导出data

[[email protected]_8_24_centos bin]# ./elasticdump --input=http://10.2.3.159:9200/reconciliation  --output=http://10.2.100.24:9200/reconciliationonline --type=data
Mon, 11 Feb 2019 09:28:58 GMT | starting dump
Mon, 11 Feb 2019 09:28:58 GMT | got 100 objects from source elasticsearch (offset: 0)
Mon, 11 Feb 2019 09:29:19 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:29:19 GMT | got 100 objects from source elasticsearch (offset: 100)
Mon, 11 Feb 2019 09:29:51 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:29:51 GMT | got 100 objects from source elasticsearch (offset: 200)
Mon, 11 Feb 2019 09:30:13 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:30:13 GMT | got 100 objects from source elasticsearch (offset: 300)
Mon, 11 Feb 2019 09:30:39 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:30:39 GMT | got 100 objects from source elasticsearch (offset: 400)
Mon, 11 Feb 2019 09:31:53 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:31:53 GMT | got 100 objects from source elasticsearch (offset: 500)
Mon, 11 Feb 2019 09:32:28 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:32:28 GMT | got 100 objects from source elasticsearch (offset: 600)
Mon, 11 Feb 2019 09:33:24 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:33:24 GMT | got 100 objects from source elasticsearch (offset: 700)
Mon, 11 Feb 2019 09:34:19 GMT | sent 100 objects to destination elasticsearch, wrote 100
Mon, 11 Feb 2019 09:34:19 GMT | got 7 objects from source elasticsearch (offset: 800)
Mon, 11 Feb 2019 09:34:47 GMT | sent 7 objects to destination elasticsearch, wrote 7
Mon, 11 Feb 2019 09:34:47 GMT | got 0 objects from source elasticsearch (offset: 807)
Mon, 11 Feb 2019 09:34:47 GMT | Total Writes: 807
Mon, 11 Feb 2019 09:34:47 GMT | dump complete
[[email protected]_8_24_centos bin]# curl  10.2.100.24:9200/_cat/indices|grep reconciliationonline
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:16 --:--:--     0yellow open reconciliationonline   KnZrsU-7TmWWBs-1-dBRrw 5 1     807  0    4.3mb    4.3mb
100 32578  100 32578    0     0   2010      0  0:00:16  0:00:16 --:--:--  7769

以上是关于使用elasticdump迁移数据到新es集群的主要内容，如果未能解决你的问题，请参考以下文章

Elasticsearch数据迁移或者版本升级，如何快速提升迁移效率？

Elasticsearch数据迁移或者版本升级，如何快速提升迁移效率？

《Linux运维总结：使用elasticdump工具迁移单节点elasticsearch数据(方案一)》

elasticdump 方法迁移数据

使用Elasticsearch-dump迁移ES数据

ES 7.7.0 数据迁移