elasticsearch和mysql的对比
https://blog.csdn.net/qq_21383435/article/details/79323383
Mapping ~ Schema
master: 负责在集群范围内创建/删除索引,将分片分配给这些节点.
data: 用来保存数据和倒排索引,node.data=ture
client: 将节点配置为客户端节点，并充当负载平衡器，将传入的请求路由到集群中的不同节点。node.master和node.data设置为false
https://blog.csdn.net/sdksdk0/article/details/78469190
Index templates
索引可使用预定义的模板进行创建,这个模板称作Index templates。模板设置包括settings和mappings，通过模式匹配的方式使得多个索引重用一个模板，例如：
curl -XPUT localhost:9200/_template/template_1 -d \'
{
"template" : "te*",
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"type1" : {
"_source" : {"enabled" : false }
}
}
}
elk设计架构
参考
k8sgithub上 https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
Elasticsearch最佳实践建议将这些节点分成三个角色：
Master 节点 - 仅用于集群管理，没有数据，没有HTTP API
Client 节点 - 用于客户端使用，无数据， 使用HTTP API
Data 节点 - 旨在存储和索引数据， 没有HTTP API
分为 管理区 存储区 读写区
定制镜像
- es
0. 我把镜像推到了dockerhub: lanny/quay.io_pires_docker-elasticsearch-kubernetes:5.6.0
1. 镜像里用到的yaml: https://github.com/lannyMa/docker-elasticsearch-kubernetes
2. 可以安装自定义插件后重新build
- kibana
0. 用到这个人的这个镜像 cfontes/kibana-xpack-less:5.5.0 https://hub.docker.com/r/cfontes/kibana-xpack-less/
这个人的git(包含了elk的k8s yaml): https://github.com/cfontes/kubernetes-elasticsearch-cluster
- k8s的elk集群的yaml
https://github.com/pires/kubernetes-elasticsearch-cluster
镜像到这个地址: https://github.com/lannyMa/kubernetes-elasticsearch-cluster
内含elk 普罗 harbor的yaml,比较实用
https://github.com/cnych/k8s-repo
集群yaml里搜HTTP_ENABLE,统一打开head访问权限
es-master.yaml
- name: HTTP_ENABLE
value: "true"
- name: HTTP_CORS_ALLOW_ORIGIN
value: "*"
创建集群
创建pv
$ cat es-data_claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: opspvc
namespace: kube-ops
annotations:
volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
先创建svc类型
后依次创建 data master client kibana
访问es api
访问kibana
打开kibana,插入示例数据
PUT _bulk
{"index":{"_index":"lagou","_type":"job","_id":"1"}}
{"title":"python分布式爬虫开发","salary_min":15000,"city":"深圳","company":{"name":"腾讯","company_addr":"深圳市软件园"},"publish_date":"2017-11-11","comments":15}
{"index":{"_index":"lagou","_type":"job","_id":"2"}}
{"title":"django开发","salary_min":15000,"city":"上海","company":{"name":"阿里","company_addr":"广州市软件园"},"publish_date":"2017-11-12","comments":20}
打开head查看数据
docker run -d -v /etc/localtime:/etc/localtime --restart=always -p 9100:9100 mobz/elasticsearch-head:5
查看数据
todo:Curator(馆长) 管理索引的工具
Curator的主要用途:
-
获取索引列表
-
过滤索引,删除/快照
| name | 用途 |
|---|---|
| Alias | 别号 |
| Allocation | 分配 |
| Close | 关 |
| Cluster Routing | 群集路由 |
| Create Index | 创建索引 |
| Delete Indices | 删除索引 |
| Delete Snapshots | 删除快照 |
| forceMerge | forceMerge |
| Index Settings | 索引设置 |
| Open | 打开 |
| Reindex | 重新编制 |
| Replicas | 副本 |
| Restore | 恢复 |
| Rollover | 滚下 |
| Shrink | 收缩 |
| Snapshot | 快照 |
elk整体链条架构
es版本兼容
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
curl http://10.100.32.137:9200
{
"name" : "es-data-5c5969967-wb2b8",
"cluster_name" : "myesdb",
"cluster_uuid" : "qSps-b9dRI2ngGHBguJ44Q",
"version" : {
"number" : "6.3.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "424e937",
"build_date" : "2018-06-11T23:38:03.357887Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}