Elasticsearch杂记

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Elasticsearch杂记相关的知识,希望对你有一定的参考价值。

一、安装使用Elasticsearch

准备3个centos7节点

1、安装

1.0 各节点时间需要同步

# /usr/sbin/ntpdate s2c.time.edu.cn

1.1 由于Elasticsearch是java开发,需要先安装JDK环境

这边使用系统默认的openjdk1.7,但是还需要安装1.7的其他包组:
[[email protected] ~]# yum install java-1.7.0-openjdk-devel

设置java环境变量:
[[email protected] ~]# vim /etc/profile.d/java.sh

export JAVA_HOME=/usr

1.2 下载安装rpm包

官方站点:https://www.elastic.co/downloads/past-releases

[[email protected] ~]# wget https://https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm

安装
[[email protected] ~]# yum install elasticsearch-1.7.2.noarch.rpm

1.3 修改配置文件

[[email protected] ~]# cd /etc/elasticsearch/
[[email protected] elasticsearch]# vim elasticsearch.yml 

cluster.name: myes1     #集群中的名字

node.name: "master1.com"    #本地节点的名字

index.number_of_shards: 5   #切片数量

index.number_of_replicas: 1 #副本数量

transport.tcp.port: 9300 #集群默认端口9300

1.4 启动

[[email protected] ~]# systemctl daemon-reload

[[email protected] ~]# systemctl start elasticsearch.service

工作端口9200/tcp,集群检测端口9300/tcp已经监听

技术分享图片

1.5 定义启动第二个节点

[[email protected] ~]# yum install java-1.7.0-openjdk-devel

设置java环境变量:
[[email protected] ~]# scp /etc/profile.d/java.sh master2:/etc/profile.d/

安装:
[[email protected] ~]# scp elasticsearch-1.7.2.noarch.rpm master2:/root/

[[email protected] ~]# yum install elasticsearch-1.7.2.noarch.rpm

复制master1节点的配置修改:
[[email protected] ~]# scp /etc/elasticsearch/elasticsearch.yml master2:/etc/elasticsearch/

修改配置文件:
[[email protected] ~]# vim /etc/elasticsearch/elasticsearch.yml 

cluster.name: myes

node.name: "master2.com"

启动:
[[email protected] ~]# systemctl start elasticsearch.service

查看状态:

技术分享图片

1.6 定义第三个节点

master3配置同master2,注意集群节点名字要一样,node节点改成master3.com即可

1.7 抓包观察集×××互信息

[[email protected] ~]# yum install -y tcpdump

技术分享图片

2、 查看集群信息(只能通过curl命令与其API进行交互)

2.1 查看节点状态是否正常

[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/?preey‘
{
  "status" : 200,
  "name" : "master1.com",
  "cluster_name" : "myes",
  "version" : {
    "number" : "1.7.2",
    "build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
    "build_timestamp" : "2015-09-14T09:49:53Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"   #相当于hello world
}

2.2 cat的API显示模式

[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/‘

技术分享图片

2.2.1 显示集群节点:

[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes‘
master2.com 10.201.106.132 7 59 0.01 d m master2.com 
master1.com 10.201.106.131 5 61 0.00 d * master1.com 
master3.com 10.201.106.133 5 71 0.08 d m master3.com 

2.2.2 显示详细格式

[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes?v‘

技术分享图片

heap.percent:堆内存  ???
ram.percent:内存使用量
load:负载
node.role:节点角色
master:*符号为主节点

2.2.3 查看选项帮助

[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes?help‘

只显示需要的列:
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes?h=name,ip,port,uptime,heap,current,load‘
master2.com 10.201.106.132 9300   8.1h 0.03 
master1.com 10.201.106.131 9300 452.3d 0.00 
master3.com 10.201.106.133 9300     8h 0.02 

2.2.4 查看主节点

[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/master‘
GFN2hX0aQpySOSYqwXD8mw master1.com 10.201.106.131 master1.com 
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/master?v‘
id                     host        ip             node        
GFN2hX0aQpySOSYqwXD8mw master1.com 10.201.106.131 master1.com 

查看集群健康状态:
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/health?v‘
epoch      timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks 
1523578507 08:15:07  myes    green           3         3      0   0    0    0        0             0 

2.3.5 _cluster 专用于查看集群的子项

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/health?v‘
{"cluster_name":"myes","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0}

pretty美观方式显示

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/health?pretty‘
{
  "cluster_name" : "myes",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/health?level=cluster&pretty

3、cluster API

3.1 查看集群版本

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/state/version?pretty‘
{
  "cluster_name" : "myes",
  "version" : 7
}

3.2 查看谁是主节点

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/state/master_node?pretty‘
{
  "cluster_name" : "myes",
  "master_node" : "GFN2hX0aQpySOSYqwXD8mw"
}

查看所有节点:
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/state/nodes?pretty‘
{
  "cluster_name" : "myes",
  "nodes" : {
    "Mv0nNWAaTz6v9OD_iTinIA" : {
      "name" : "master2.com",
      "transport_address" : "inet[/10.201.106.132:9300]",
      "attributes" : { }
    },
    "GFN2hX0aQpySOSYqwXD8mw" : {
      "name" : "master1.com",
      "transport_address" : "inet[/10.201.106.131:9300]",
      "attributes" : { }
    },
    "YAY5iFmJQAGHa6OuvLaMuA" : {
      "name" : "master3.com",
      "transport_address" : "inet[/10.201.106.133:9300]",
      "attributes" : { }
    }
  }
}

3.3 查看集群统计数据

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/stats/?pretty‘

4、节点状态信息

4.1 查看节点信息

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_nodes/stats/?pretty‘

5、Plusins(插件)

5.1 查看ES(Elasticsearch)插件目录

[[email protected] ~]# rpm -ql elasticsearch | grep plugin
/usr/share/elasticsearch/bin/plugin
/usr/share/elasticsearch/plugins    #插件目录

5.2 列出所有插件

查看使用帮助:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -h

[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
    - No plugin detected in /usr/share/elasticsearch/plugins

5.3 安装插件

直接通过URL安装插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i marvel -u http://10.201.106.1:8080/plugins/marvel-latest.zip
-> Installing marvel...
Trying http://10.201.106.1:8080/plugins/marvel-latest.zip...
Downloading .............................DONE
Installed marvel into /usr/share/elasticsearch/plugins/marvel

列出安装的插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
    - marvel

本地安装方法:
下载目录文件(不太好用):
wget -c -r -np -k -L -p http://10.201.106.1:8080/plugins/    

[[email protected] plugins]# pwd
/root/10.201.106.1:8080/plugins
[[email protected] plugins]# ls
bigdesk-latest.zip             index.html
elasticsearch-head-latest.zip  ~login
elasticsearch-kopf-master.zip  marvel-latest.zip
将以上文件mv到root家目录下

本地安装bigdesk插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i bigdesk -u file:///root/bigdesk-latest.zip

查看:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
    - marvel
    - bigdesk

安装其他插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i head -u file:///root/elasticsearch-head-latest.zip

[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i kopf -u file:///root/elasticsearch-kopf-master.zip

5.4 访问插件

http://10.201.106.131:9200/_plugin/marvel

技术分享图片

http://10.201.106.131:9200/_plugin/bigdesk

技术分享图片

http://10.201.106.131:9200/_plugin/head

技术分享图片

http://10.201.106.131:9200/_plugin/kopf

技术分享图片

6、索引

6.1 查看刚才插件安装自动生成的索引

[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cat/indices?v‘
health status index              pri rep docs.count docs.deleted store.size pri.store.size 
green  open   .marvel-2018.04.13   1   1        435            0      3.9mb          2.1mb 
green  open   .marvel-kibana       1   1          1            0      6.5kb          3.2kb 

6.2 创建索引下的文档

郭靖同学,性别男,年龄25,课程:降龙十八掌
[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class1/1?pretty‘ -d ‘
> {
> "first_name":"Jing",
> "last_name":"Guo",
> "gender":"Male",
> "age":25,
> "courses":"Xianglong Shiba Zhang"
> }‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "1",
  "_version" : 1,
  "created" : true
}

添加第二个同学:
[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class1/2?pretty‘ -d ‘
> {
> "first_name":"Rong",
> "last_name":"Huang",
> "gender":"Female",
> "age":23,
> "courses":"Luoying Shenjian"
> }‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "2",
  "_version" : 1,
  "created" : true
}

6.3 显示刚才创建的索引下的文档

[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/1?pretty‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source":
{
"first_name":"Jing",
"last_name":"Guo",
"gender":"Male",
"age":25,
"courses":"Xianglong Shiba Zhang"
}
}

查看第二个同学:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/2?pretty‘

put重复ID,会被替换。建议生产环境,使用ID生成器

6.4 更新文档

PUT方法会覆盖原有文档:PUT方法会覆盖原有文档:

修改黄蓉的年龄:
[[email protected] ~]# curl -XPOST ‘localhost:9200/students/class1/2/_update?pretty‘ -d ‘
{
"doc":{"age":22}
}‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "2",
  "_version" : 2
}

查看:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/2?pretty‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "2",
  "_version" : 2,
  "found" : true,
  "_source":{"first_name":"Rong","last_name":"Huang","gender":"Female","age":22,"courses":"Luoying Shenjian"}

6.5 删除文档

[[email protected] ~]# curl -XDELETE ‘localhost:9200/students/class1/2‘
{"found":true,"_index":"students","_type":"class1","_id":"2","_version":3}[[email protected] ~]# 

查看已经没找到ID2的文档信息:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/2?pretty‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "2",
  "found" : false
}

6.6 删除整个索引

查看有哪些索引:
[[email protected] ~]# curl -XGET ‘localhost:9200/_cat/indices?v‘
health status index              pri rep docs.count docs.deleted store.size pri.store.size 
green  open   .marvel-2018.04.13   1   1       4942            0     31.5mb           16mb 
green  open   students             5   1          1            0      7.8kb          3.9kb 
green  open   .marvel-kibana       1   1          1            0      6.5kb          3.2kb 

删除students索引:
[[email protected] ~]# curl -XDELETE ‘localhost:9200/students‘
{"acknowledged":true}

再查看索引,students已经没了:
[[email protected] ~]# curl -XGET ‘localhost:9200/_cat/indices?v‘

6.7 重新创建文档

[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class1/1?pretty‘ -d ‘
> {
> "first_name":"Jing",
> "last_name":"Guo",
> "gender":"Male",
> "age":25,
> "courses":"Xianglong Shiba Zhang"
> }‘

[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class/2?pretty‘ -d ‘
> {
> "first_name":"Rong",
> "last_name":"Huang",
> "gender":"Female",
> "age":23,
> "courses":"Luoying shenjian"
> }‘

7、数据查询

7.1 列出索引下的所有文档

全量查询,较少使用
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?pretty‘

qeury body查询方式,能编写更为复杂的查询:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?pretty‘ -d ‘
> {
> "query": { "match_all": {} }
> }‘

7.2 查询所有索引

数据特别多
[[email protected] ~]# curl -XGET ‘localhost:9200/_search?pretty‘

/_search:所有索引;
/INDEX_NAME/_search:单索引;
/INDEX1,INDEX2/_search:多索引;
/s*,t*/_search:通配符查找索引
/students/class1/_search:单类型搜索
/students/class1,class2/_search:多类型搜索

7.3 几种查询方式

1、GET /_search?q=‘Xianglong‘
2、GET /_search?q=‘Xianglong Shiba Zhang‘
3、GET /_search?q=courses:‘Xianglong Shibao Zhang‘
4、GET /_search?q=courses:‘Xianglong‘

前两个:表示在_all域搜索;
后两个:在指定的域上搜索;

7.4 几种查询方式测试

删除
[[email protected] ~]# curl -XDELETE ‘localhost:9200/students/class/1‘

[[email protected] ~]# curl -XDELETE ‘localhost:9200/students/class/2‘
{"found":true,"_index":"students","_type":"class","_id":"2","_version":2}[[email protected] ~]# 

添加:
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/1?pretty‘ -d ‘
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}‘

[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/2?pretty‘ -d ‘
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "2",
  "_version" : 1,
  "created" : true
}

查询:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q="Guo"‘
{"took":36,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.11506981,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"

添加
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/3?pretty‘ -d ‘
{
"name": "Huang Rong",
"gender": "Female",
"age": 22,
"parent": "Guo Xiaotian"
}‘
{
  "_index" : "students",
  "_type" : "class1",
  "_id" : "3",
  "_version" : 1,
  "created" : true
}

只要域中带guo,都能搜出来:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q="Guo"‘
{"took":12,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":3,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.11506981,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}},{"_index":"students","_type":"class1","_id":"3","_score":0.11506981,"_source":
{
"name": "Huang Rong",
"gender": "Female",
"age": 22,
"parent": "Guo Xiaotian"
}}]}}[

只搜索Guo Jing
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q="Guo%20Jing"‘{"took":11,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.23013961,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.23013961,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"

查询name=Guo
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=name:"Guo"‘
{"took":30,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.19178301,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.19178301,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.19178301,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}}]}}[

再添加一个人: 
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/4?pretty‘ -d ‘
> {
> "name": "GuoXiang",
> "gender": "Female",
> "age": 10,
> "class": "Emei Pai"
> }‘

再次搜索Guo,没有郭襄(切词后的结果):
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=name:"Guo"‘
{"took":18,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.19178301,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.19178301,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.19178301,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}}]}}

只搜Guo Jing:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=name:"Guo%20Jing"‘
{"took":21,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.38356602,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.38356602,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}}]}}[[email protected] ~]# 

7.5 年龄搜索测试

[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=25‘
{"took":10,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}}]}}

再加一个人,声明25人之一
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/5‘ -d ‘
> {
> "name": "Xiaolong Nv",
> "gender": "Female",
> "age": 18,
> "description": "one of 25"
> }‘

再搜索25:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=25‘
{"took":12,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"5","_score":0.11506981,"_source":
{
"name": "Xiaolong Nv",
"gender": "Female",
"age": 18,
"description": "one of 25"
}},{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}}]}}[[email protected] ~]# 

7.6 通过marvel插件链接插件

浏览器访问
http://10.201.106.131:9200/_plugin/marvel/sense/

8、映射

8.1 查看指定类型的mapping

[[email protected] ~]# curl ‘localhost:9200/students/_mapping/class1?pretty‘
{
  "students" : {
    "mappings" : {
      "class1" : {
        "properties" : {
          "age" : {
            "type" : "long"
          },
          "class" : {
            "type" : "string"
          },
          "courses" : {
            "type" : "string"
          },
          "description" : {
            "type" : "string"
          },
          "first_name" : {
            "type" : "string"
          },
          "gender" : {
            "type" : "string"
          },
          "last_name" : {
            "type" : "string"
          },
          "name" : {
            "type" : "string"
          },
          "parent" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

以上是关于Elasticsearch杂记的主要内容,如果未能解决你的问题,请参考以下文章

ElasticSearch学习问题记录——Invalid shift value in prefixCoded bytes (is encoded value really an INT?)(代码片段

logstash&Kibana杂记

Elasticsearch笔记九之优化

使用标准库Ruby将数据标记到Elasticsearch批量中

技术杂记

杂记之--苹果4s手机呼叫转移怎么设置