elk6.3 centos集群搭建 head插件安装

Posted 2020-12-25 学无终

版本elk均为6.3+centos7.0

准备工作

官网下载elk6.3的linux环境的压缩包，sftp上传

下载对应的head插件sftp上传到指定目录

tar.gz文件解压

tar -zxvf filename -C dirname

zip文件

安装zip软件

unzip filename -d dirname

1.root用户不能启动elasticsearch

 

参考：http://www.cnblogs.com/ding2016/p/6879584.html

https://www.cnblogs.com/Onlywjy/p/Elasticsearch.html

2.创建用户

useradd fan

修改密码

passwd fan

 

3.给用户root权限（测试图简单，线上请自行建组并授权）

赋予用户fan root权限

chmod -v u+w /etc/sudoers

visudo

在root那行添加如下

## Allow root to run any commands anywhere

root ALL=(ALL) ALL

fan ALL=(ALL) ALL

4.给目录授权 chmod 777 -R dir

 

 

5.修改配置文件limits.conf

vim /etc/security/limits.conf

#* soft core 0

#* hard rss 10000

#@student hard nproc 20

#@faculty soft nproc 20

#@faculty hard nproc 50

#ftp hard nproc 0

#@student - maxlogins 4

fan hard nofile 65536

fan soft nofile 65536

"/etc/security/limits.conf" 62L, 2465C 已写入

fan为启动用户，自己根据自己用户名更改，不改改配置会报如下错误

[2018-04-29T07:10:31,207][INFO ][o.e.t.TransportService ] [node_01] publish_address {192.168.2.10:9300}, bound_addresses {[::]:9300}

[2018-04-29T07:10:31,222][INFO ][o.e.b.BootstrapChecks ] [node_01] bound or publishing to a non-loopback address, enforcing bootstrap checks

ERROR: [2] bootstrap checks failed

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

 

 

 

6.修改配置文件sysctl.conf

vi /etc/sysctl.conf

文件末尾添加

vm.max_map_count=655360

否则启动报如下错误

[2018-04-29T07:19:56,208][INFO ][o.e.b.BootstrapChecks ] [node_01] bound or publishing to a non-loopback address, enforcing bootstrap checks

ERROR: [1] bootstrap checks failed

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

7.修改elasticsearch配置文件elasticsearch.yml

#cluster.name: my-application

cluster.name: elk

#

# ------------------------------------ Node ------------------------------------

#

# Use a descriptive name for the node:

#

node.name: node_01

#指定该节点是否有资格被选举成为node，默认是true，es是默认集群中的第一台机器为master，如果这台机挂了就会重新选举

node.master: true

#指定该节点是否存储索引数据，默认为true

node.data: true

#设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1，对于大的集群来说，可以设置大一点的值（2-4）

discovery.zen.minimum_master_nodes: 1

#设置节点间交互的tcp端口，默认是9300

transport.tcp.port: 9300

#

# Add custom attributes to the node:

#

#node.attr.rack: r1

#

# ----------------------------------- Paths ------------------------------------

#

# Path to directory where to store the data (separate multiple locations by comma):

#

#path.data: /path/to/data

path.data: /path/to/data

#

# Path to log files:

#

#path.logs: /path/to/logs

path.logs: /path/to/logs

#

# ----------------------------------- Memory -----------------------------------

#

# Lock the memory on startup:

#

#bootstrap.memory_lock: true

#

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

#

# Elasticsearch performs poorly when the system is swapping the memory.

#

# ---------------------------------- Network -----------------------------------

#

# Set the bind address to a specific IP (IPv4 or IPv6):

#

#network.host: 192.168.0.1

network.host: 192.168.2.10

#

# Set a custom port for HTTP:

#

#http.port: 9200

http.port: 9200

#

# For more information, consult the network module documentation.

#

# --------------------------------- Discovery ----------------------------------

#

# Pass an initial list of hosts to perform discovery when new node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

#

#discovery.zen.ping.unicast.hosts: ["host1", "host2"]

discovery.zen.ping.unicast.hosts: ["192.168.2.10", "192.168.2.11", "192.168.2.12"]

 

#

# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

#

#discovery.zen.minimum_master_nodes:

#

# For more information, consult the zen discovery module documentation.

#

# ---------------------------------- Gateway -----------------------------------

#

# Block initial recovery after a full cluster restart until N nodes are started:

#

#gateway.recover_after_nodes: 3

#

# For more information, consult the gateway module documentation.

#

# ---------------------------------- Various -----------------------------------

#

# Require explicit names when deleting indices:

#

#action.destructive_requires_name: true

#

#head插件

http.cors.enabled: true

http.cors.allow-origin: "*"

最后两行开始不要加，后面配置head插件后再添加

 

8.elasticsearch启动成功

[fan@master elasticsearch-6.3.0]$ jps

114672 Jps

114595 Elasticsearch

 

elasticsearch.yml

参考：https://blog.csdn.net/lu_wei_wei/article/details/51263153

https://www.cnblogs.com/zlslch/p/6419948.html

 

9.关闭防火墙

“failed to obtain node locks”

jps

将elasticsearch kill

elasticsearch 没有到主机的路由

关闭防火墙

 

集群访问：

http://192.168.2.11:9200/_cluster/state/nodes?pretty

参考：https://www.cnblogs.com/zhi-leaf/p/8484337.html

 

10.logstash 同elasticsearch解压，然后修改配置文件

logstash 采集后台

logstash.conf

自己根据实际情况更改input中的path和output中的host index

11.添加日志文件测试

logstash 后台会有添加记录

12.下载解压kibana 同上面elasticsearch

修改conf 下 kibana.yml添加elasticsearch地址，添加一个即可

elasticsearch.url: "http://192.168.2.10:9200"

然后后台启动bin目录下 nohup ./kibana &

 

13.head安装

前提条件

a.下载master-head插件，https://github.com/mobz/elasticsearch-head

上面地址下载master.zip上传解压

也可以命令：wget https://github.com/mobz/elasticsearch-head/archive/master.zip

我是下载的master.zip所以要解压

安装zip软件

yum install -y zip unzip

解压master.zip

unzip master.zip -d /srv/

b.node下载安装

下载地址：https://npm.taobao.org/mirrors/node/

下载后解压同elasticsearch

配置环境变量

vim /etc/profile

export NODE_HOME=/srv/node-v4.4.7-linux-x64

export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$NODE_HOME/bin:$PATH

生效环境变量source /etc/profile

npm install -g grunt-cli

 

[root@master elasticsearch-head-master]# grunt -version

grunt-cli v1.2.0

修改 Gruntfile.js

vim Gruntfile.js

connect: {

server: {

options: {

port: 9100,

base: \'.\',

keepalive: true,

hostname: \'*\'

}

}

}

npm install

启动head插件

grunt server

访问http://192.168.2.10:9100/

参考：http://www.cnblogs.com/Onlywjy/p/Elasticsearch.html

14.访问kibana

http://192.168.2.10:5601

我这边直接拷贝了一个日志文件放到了前面logstash中配置的路径下

elk搭建完毕

 

Solr和Es区别

Solr:查询多且查询的时候更新较少。支持word, pdf等富文本索引开发者社区完善,稳定性高一点。但是笨重

1Es:一边查一边更新多系统。只支持ison格式。轻便,集群部署简单。实时性高。

 

由于环境不同大家可能遇到其他的问题，有问题可以加入qq群：513650703共同交流学习