Elasticsearch杂记
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Elasticsearch杂记相关的知识,希望对你有一定的参考价值。
一、安装使用Elasticsearch
准备3个centos7节点
1、安装
1.0 各节点时间需要同步
# /usr/sbin/ntpdate s2c.time.edu.cn
1.1 由于Elasticsearch是java开发,需要先安装JDK环境
这边使用系统默认的openjdk1.7,但是还需要安装1.7的其他包组:
[[email protected] ~]# yum install java-1.7.0-openjdk-devel
设置java环境变量:
[[email protected] ~]# vim /etc/profile.d/java.sh
export JAVA_HOME=/usr
1.2 下载安装rpm包
官方站点:https://www.elastic.co/downloads/past-releases
[[email protected] ~]# wget https://https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.noarch.rpm
安装
[[email protected] ~]# yum install elasticsearch-1.7.2.noarch.rpm
1.3 修改配置文件
[[email protected] ~]# cd /etc/elasticsearch/
[[email protected] elasticsearch]# vim elasticsearch.yml
cluster.name: myes1 #集群中的名字
node.name: "master1.com" #本地节点的名字
index.number_of_shards: 5 #切片数量
index.number_of_replicas: 1 #副本数量
transport.tcp.port: 9300 #集群默认端口9300
1.4 启动
[[email protected] ~]# systemctl daemon-reload
[[email protected] ~]# systemctl start elasticsearch.service
工作端口9200/tcp,集群检测端口9300/tcp已经监听
1.5 定义启动第二个节点
[[email protected] ~]# yum install java-1.7.0-openjdk-devel
设置java环境变量:
[[email protected] ~]# scp /etc/profile.d/java.sh master2:/etc/profile.d/
安装:
[[email protected] ~]# scp elasticsearch-1.7.2.noarch.rpm master2:/root/
[[email protected] ~]# yum install elasticsearch-1.7.2.noarch.rpm
复制master1节点的配置修改:
[[email protected] ~]# scp /etc/elasticsearch/elasticsearch.yml master2:/etc/elasticsearch/
修改配置文件:
[[email protected] ~]# vim /etc/elasticsearch/elasticsearch.yml
cluster.name: myes
node.name: "master2.com"
启动:
[[email protected] ~]# systemctl start elasticsearch.service
查看状态:
1.6 定义第三个节点
master3配置同master2,注意集群节点名字要一样,node节点改成master3.com即可
1.7 抓包观察集×××互信息
[[email protected] ~]# yum install -y tcpdump
2、 查看集群信息(只能通过curl命令与其API进行交互)
2.1 查看节点状态是否正常
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/?preey‘
{
"status" : 200,
"name" : "master1.com",
"cluster_name" : "myes",
"version" : {
"number" : "1.7.2",
"build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
"build_timestamp" : "2015-09-14T09:49:53Z",
"build_snapshot" : false,
"lucene_version" : "4.10.4"
},
"tagline" : "You Know, for Search" #相当于hello world
}
2.2 cat的API显示模式
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/‘
2.2.1 显示集群节点:
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes‘
master2.com 10.201.106.132 7 59 0.01 d m master2.com
master1.com 10.201.106.131 5 61 0.00 d * master1.com
master3.com 10.201.106.133 5 71 0.08 d m master3.com
2.2.2 显示详细格式
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes?v‘
heap.percent:堆内存 ???
ram.percent:内存使用量
load:负载
node.role:节点角色
master:*符号为主节点
2.2.3 查看选项帮助
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes?help‘
只显示需要的列:
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/nodes?h=name,ip,port,uptime,heap,current,load‘
master2.com 10.201.106.132 9300 8.1h 0.03
master1.com 10.201.106.131 9300 452.3d 0.00
master3.com 10.201.106.133 9300 8h 0.02
2.2.4 查看主节点
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/master‘
GFN2hX0aQpySOSYqwXD8mw master1.com 10.201.106.131 master1.com
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/master?v‘
id host ip node
GFN2hX0aQpySOSYqwXD8mw master1.com 10.201.106.131 master1.com
查看集群健康状态:
[[email protected] ~]# curl -X GET ‘http://10.201.106.131:9200/_cat/health?v‘
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks
1523578507 08:15:07 myes green 3 3 0 0 0 0 0 0
2.3.5 _cluster 专用于查看集群的子项
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/health?v‘
{"cluster_name":"myes","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0}
pretty美观方式显示
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/health?pretty‘
{
"cluster_name" : "myes",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/health?level=cluster&pretty‘
3、cluster API
3.1 查看集群版本
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/state/version?pretty‘
{
"cluster_name" : "myes",
"version" : 7
}
3.2 查看谁是主节点
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/state/master_node?pretty‘
{
"cluster_name" : "myes",
"master_node" : "GFN2hX0aQpySOSYqwXD8mw"
}
查看所有节点:
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/state/nodes?pretty‘
{
"cluster_name" : "myes",
"nodes" : {
"Mv0nNWAaTz6v9OD_iTinIA" : {
"name" : "master2.com",
"transport_address" : "inet[/10.201.106.132:9300]",
"attributes" : { }
},
"GFN2hX0aQpySOSYqwXD8mw" : {
"name" : "master1.com",
"transport_address" : "inet[/10.201.106.131:9300]",
"attributes" : { }
},
"YAY5iFmJQAGHa6OuvLaMuA" : {
"name" : "master3.com",
"transport_address" : "inet[/10.201.106.133:9300]",
"attributes" : { }
}
}
}
3.3 查看集群统计数据
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cluster/stats/?pretty‘
4、节点状态信息
4.1 查看节点信息
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_nodes/stats/?pretty‘
5、Plusins(插件)
5.1 查看ES(Elasticsearch)插件目录
[[email protected] ~]# rpm -ql elasticsearch | grep plugin
/usr/share/elasticsearch/bin/plugin
/usr/share/elasticsearch/plugins #插件目录
5.2 列出所有插件
查看使用帮助:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -h
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
- No plugin detected in /usr/share/elasticsearch/plugins
5.3 安装插件
直接通过URL安装插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i marvel -u http://10.201.106.1:8080/plugins/marvel-latest.zip
-> Installing marvel...
Trying http://10.201.106.1:8080/plugins/marvel-latest.zip...
Downloading .............................DONE
Installed marvel into /usr/share/elasticsearch/plugins/marvel
列出安装的插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
- marvel
本地安装方法:
下载目录文件(不太好用):
wget -c -r -np -k -L -p http://10.201.106.1:8080/plugins/
[[email protected] plugins]# pwd
/root/10.201.106.1:8080/plugins
[[email protected] plugins]# ls
bigdesk-latest.zip index.html
elasticsearch-head-latest.zip ~login
elasticsearch-kopf-master.zip marvel-latest.zip
将以上文件mv到root家目录下
本地安装bigdesk插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i bigdesk -u file:///root/bigdesk-latest.zip
查看:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -l
Installed plugins:
- marvel
- bigdesk
安装其他插件:
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i head -u file:///root/elasticsearch-head-latest.zip
[[email protected] ~]# /usr/share/elasticsearch/bin/plugin -i kopf -u file:///root/elasticsearch-kopf-master.zip
5.4 访问插件
http://10.201.106.131:9200/_plugin/marvel
http://10.201.106.131:9200/_plugin/bigdesk
http://10.201.106.131:9200/_plugin/head
http://10.201.106.131:9200/_plugin/kopf
6、索引
6.1 查看刚才插件安装自动生成的索引
[[email protected] ~]# curl -XGET ‘http://10.201.106.131:9200/_cat/indices?v‘
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open .marvel-2018.04.13 1 1 435 0 3.9mb 2.1mb
green open .marvel-kibana 1 1 1 0 6.5kb 3.2kb
6.2 创建索引下的文档
郭靖同学,性别男,年龄25,课程:降龙十八掌
[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class1/1?pretty‘ -d ‘
> {
> "first_name":"Jing",
> "last_name":"Guo",
> "gender":"Male",
> "age":25,
> "courses":"Xianglong Shiba Zhang"
> }‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "1",
"_version" : 1,
"created" : true
}
添加第二个同学:
[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class1/2?pretty‘ -d ‘
> {
> "first_name":"Rong",
> "last_name":"Huang",
> "gender":"Female",
> "age":23,
> "courses":"Luoying Shenjian"
> }‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "2",
"_version" : 1,
"created" : true
}
6.3 显示刚才创建的索引下的文档
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/1?pretty‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "1",
"_version" : 1,
"found" : true,
"_source":
{
"first_name":"Jing",
"last_name":"Guo",
"gender":"Male",
"age":25,
"courses":"Xianglong Shiba Zhang"
}
}
查看第二个同学:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/2?pretty‘
put重复ID,会被替换。建议生产环境,使用ID生成器
6.4 更新文档
PUT方法会覆盖原有文档:PUT方法会覆盖原有文档:
修改黄蓉的年龄:
[[email protected] ~]# curl -XPOST ‘localhost:9200/students/class1/2/_update?pretty‘ -d ‘
{
"doc":{"age":22}
}‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "2",
"_version" : 2
}
查看:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/2?pretty‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "2",
"_version" : 2,
"found" : true,
"_source":{"first_name":"Rong","last_name":"Huang","gender":"Female","age":22,"courses":"Luoying Shenjian"}
6.5 删除文档
[[email protected] ~]# curl -XDELETE ‘localhost:9200/students/class1/2‘
{"found":true,"_index":"students","_type":"class1","_id":"2","_version":3}[[email protected] ~]#
查看已经没找到ID2的文档信息:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/class1/2?pretty‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "2",
"found" : false
}
6.6 删除整个索引
查看有哪些索引:
[[email protected] ~]# curl -XGET ‘localhost:9200/_cat/indices?v‘
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open .marvel-2018.04.13 1 1 4942 0 31.5mb 16mb
green open students 5 1 1 0 7.8kb 3.9kb
green open .marvel-kibana 1 1 1 0 6.5kb 3.2kb
删除students索引:
[[email protected] ~]# curl -XDELETE ‘localhost:9200/students‘
{"acknowledged":true}
再查看索引,students已经没了:
[[email protected] ~]# curl -XGET ‘localhost:9200/_cat/indices?v‘
6.7 重新创建文档
[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class1/1?pretty‘ -d ‘
> {
> "first_name":"Jing",
> "last_name":"Guo",
> "gender":"Male",
> "age":25,
> "courses":"Xianglong Shiba Zhang"
> }‘
[[email protected] ~]# curl -XPUT ‘10.201.106.131:9200/students/class/2?pretty‘ -d ‘
> {
> "first_name":"Rong",
> "last_name":"Huang",
> "gender":"Female",
> "age":23,
> "courses":"Luoying shenjian"
> }‘
7、数据查询
7.1 列出索引下的所有文档
全量查询,较少使用
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?pretty‘
qeury body查询方式,能编写更为复杂的查询:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?pretty‘ -d ‘
> {
> "query": { "match_all": {} }
> }‘
7.2 查询所有索引
数据特别多
[[email protected] ~]# curl -XGET ‘localhost:9200/_search?pretty‘
/_search:所有索引;
/INDEX_NAME/_search:单索引;
/INDEX1,INDEX2/_search:多索引;
/s*,t*/_search:通配符查找索引
/students/class1/_search:单类型搜索
/students/class1,class2/_search:多类型搜索
7.3 几种查询方式
1、GET /_search?q=‘Xianglong‘
2、GET /_search?q=‘Xianglong Shiba Zhang‘
3、GET /_search?q=courses:‘Xianglong Shibao Zhang‘
4、GET /_search?q=courses:‘Xianglong‘
前两个:表示在_all域搜索;
后两个:在指定的域上搜索;
7.4 几种查询方式测试
删除
[[email protected] ~]# curl -XDELETE ‘localhost:9200/students/class/1‘
[[email protected] ~]# curl -XDELETE ‘localhost:9200/students/class/2‘
{"found":true,"_index":"students","_type":"class","_id":"2","_version":2}[[email protected] ~]#
添加:
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/1?pretty‘ -d ‘
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}‘
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/2?pretty‘ -d ‘
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "2",
"_version" : 1,
"created" : true
}
查询:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q="Guo"‘
{"took":36,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.11506981,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
添加
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/3?pretty‘ -d ‘
{
"name": "Huang Rong",
"gender": "Female",
"age": 22,
"parent": "Guo Xiaotian"
}‘
{
"_index" : "students",
"_type" : "class1",
"_id" : "3",
"_version" : 1,
"created" : true
}
只要域中带guo,都能搜出来:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q="Guo"‘
{"took":12,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":3,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.11506981,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}},{"_index":"students","_type":"class1","_id":"3","_score":0.11506981,"_source":
{
"name": "Huang Rong",
"gender": "Female",
"age": 22,
"parent": "Guo Xiaotian"
}}]}}[
只搜索Guo Jing
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q="Guo%20Jing"‘{"took":11,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.23013961,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.23013961,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
查询name=Guo
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=name:"Guo"‘
{"took":30,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.19178301,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.19178301,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.19178301,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}}]}}[
再添加一个人:
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/4?pretty‘ -d ‘
> {
> "name": "GuoXiang",
> "gender": "Female",
> "age": 10,
> "class": "Emei Pai"
> }‘
再次搜索Guo,没有郭襄(切词后的结果):
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=name:"Guo"‘
{"took":18,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.19178301,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.19178301,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}},{"_index":"students","_type":"class1","_id":"2","_score":0.19178301,"_source":
{
"name": "Yang Guo",
"gender": "Male",
"age": 17,
"class": "Gumu Pai"
}}]}}
只搜Guo Jing:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=name:"Guo%20Jing"‘
{"took":21,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.38356602,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.38356602,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}}]}}[[email protected] ~]#
7.5 年龄搜索测试
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=25‘
{"took":10,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":1,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}}]}}
再加一个人,声明25人之一
[[email protected] ~]# curl -XPUT ‘localhost:9200/students/class1/5‘ -d ‘
> {
> "name": "Xiaolong Nv",
> "gender": "Female",
> "age": 18,
> "description": "one of 25"
> }‘
再搜索25:
[[email protected] ~]# curl -XGET ‘localhost:9200/students/_search?q=25‘
{"took":12,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":0.11506981,"hits":[{"_index":"students","_type":"class1","_id":"5","_score":0.11506981,"_source":
{
"name": "Xiaolong Nv",
"gender": "Female",
"age": 18,
"description": "one of 25"
}},{"_index":"students","_type":"class1","_id":"1","_score":0.11506981,"_source":
{
"name": "Guo Jing",
"gender": "Male",
"age": 25,
"class": "Gai Bang"
}}]}}[[email protected] ~]#
7.6 通过marvel插件链接插件
浏览器访问
http://10.201.106.131:9200/_plugin/marvel/sense/
8、映射
8.1 查看指定类型的mapping
[[email protected] ~]# curl ‘localhost:9200/students/_mapping/class1?pretty‘
{
"students" : {
"mappings" : {
"class1" : {
"properties" : {
"age" : {
"type" : "long"
},
"class" : {
"type" : "string"
},
"courses" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"first_name" : {
"type" : "string"
},
"gender" : {
"type" : "string"
},
"last_name" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"parent" : {
"type" : "string"
}
}
}
}
}
}
以上是关于Elasticsearch杂记的主要内容,如果未能解决你的问题,请参考以下文章
ElasticSearch学习问题记录——Invalid shift value in prefixCoded bytes (is encoded value really an INT?)(代码片段