ELK集群搭建简略记录
Posted moakia
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ELK集群搭建简略记录相关的知识，希望对你有一定的参考价值。
ELK集群搭建简略记录
1.规划
三台主机搭建集群，对应关系如下
10.0.0.4 node1
10.0.0.8 node2
10.0.0.9 node3
2.添加host（各节点）
cat >> /etc/hosts <<EOF
10.0.0.4 node1
10.0.0.8 node2
10.0.0.9 node3
EOF
3.下载安装elk（各节点）
mkdir /elastic
mkdir -p /data/elasticsearch/{data,logs}wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.2-linux-x86_64.tar.gz
tar -xzf elasticsearch-7.15.2-linux-x86_64.tar.gz -C /elastic
mv /elastic/elasticsearch-7.15.2 /elastic/elasticsearch
cd /elastic/elasticsearch
useradd elastic
chown -R elastic:elastic /elastic
chown -R elastic:elastic /data/elasticsearch
3.修改jvm.options（各节点）
#参考(java最大只能分配32G内存，建议分配为内存的一半)
-Xms4g
-Xmx4g
4.修改使用elk自带的jdk（各节点）
#修改使用自带的java虚拟机,修改/elastic/elasticsearch/bin/elasticsearch-env配置#在if [ ! -z "$ES_JAVA_HOME" ]; then前添加配置
sed -i / "$ES_JAVA_HOME/iES_JAVA_HOME=/elastic/elasticsearch/jdk/ /elastic/elasticsearch/bin/elasticsearch-env
5.生成集群间通信的SSL证书（各节点）
#只有配置这个才能使用用户认证#生成证书,记得拷贝到其他节点,如果要拷贝其他节点，记得添加ipsan;如果不添加，把证书拷贝到其他节点，单独生成证书 #先通过elasticsearch-certutil ca先生成公钥 #再通过elasticsearch-certutil cert 生成证书
cd /elastic/elasticsearch/
/elastic/elasticsearch/bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip "10.0.0.4","10.0.0.8","10.0.0.9"
cd /elastic/elasticsearch/
mkdir /elastic/elasticsearch/config/certs
mv elastic-certificates.p12 /elastic/elasticsearch/config/certs
chown -R elastic:elastic /elastic/elasticsearch/config/certs
scp xxx nodex:xxx
6.生成用于客户端访问的SSL证书（各节点）
#当然也可以复用tls生成的#这个命令不仅会生成node节点的证书和配置文件，也会生成kibana所需要的配置 #具体参考 https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#如果麻烦可以不启用
cd /elastic/elasticsearch/
/elastic/elasticsearch/bin/elasticsearch-certutil http
scp xxx node1:xxxx
7.集群配置文件参考
#集群名称
cluster.name: bl-els
# 节点名称，3个节点对应名称["node-1","node-2","node-3"]
node.name: node-1
# ES数据存储路径
path.data: /data/elasticsearch/data
# ES日志存储路径
path.logs: /data/elasticsearch/logs
# 锁定内存
bootstrap.memory_lock: true
# HTTP访问IP，内网IP、外网IP都可以访问
network.host: 0.0.0.0
# HTTP访问端口
http.port: 9200
# 集群访问端口
transport.tcp.port: 9300
# 种子节点的地址列表
discovery.seed_hosts: ["node1", "node2", "node3"]
# 可以成为主节点的名称列表
cluster.initial_master_nodes: ["node1", "node2", "node3"]
#设置elk 账号和密码登录
#节点间安全通信，只有配置这个才能配置用户认证
#elasticsearch-certutil ca cert 之类的命令生成
#参考 https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup.html#generate-certificates
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12
#接口使用https通信(视情况使用)面向客户端，面向浏览器 可以使用 elasticsearch-certutil http生成；配置完成后需要在kinaba启用https
#参考https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /elastic/elasticsearch/config/certs/elastic-certificates.p12
8.调整系统参数配置
echo "vm.max_map_count = 655300" >>/etc/sysctl.conf
echo "* - memlock unlimited" >>/etc/security/limits.conf
echo "* - nofile 655360" >>/etc/security/limits.conf
echo "* - as unlimited" >>/etc/security/limits.conf
echo "* - nproc 2056474" >>/etc/security/limits.conf
echo "* - fsize unlimited" >>/etc/security/limits.conf
echo "net.ipv4.tcp_abort_on_overflow = 1" >>/etc/sysctl.conf
echo "net.core.somaxconn = 2048" >>/etc/sysctl.conf
sysctl -p
9.调整systemd相关参数
cat >> /etc/systemd/system.conf << EOF
DefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity
EOF
10.设置开机启动-initd方式
cat > /etc/init.d/elasticsearch <<EOF
#!/bin/sh
#chkconfig: 2345 80 05
#description: elasticsearch
els_user=elastic
els_menu=/elastic/elasticsearch
case "$1" in
start)
su $els_user<<!
cd $els_menu
./bin/elasticsearch -d
exit
!
echo "elasticsearch startup"
;;
stop)
els_pid=`ps aux | grep org.elasticsearch.bootstrap.Elasticsearch | grep -v grep | awk {print $2}`kill $els_pid
echo "elasticsearch stopped"
;;
restart)
els_pid=`ps aux | grep org.elasticsearch.bootstrap.Elasticsearch | grep -v grep | awk {print $2}`kill $els_pid
echo "elasticsearch stopped"
su $els_user<<!
cd $els_menu
./bin/elasticsearch -d
exit
!
echo "elasticsearch startup"
;;
*)
echo "start|stop|restart"
;;
esac
exit $?
EOF
chmod +x /etc/init.d/elasticsearch
chkconfig --add elasticsearch
11.设置开机启动-systemd方式
cat > /lib/systemd/system/elasticsearch.service <<EOF
[Unit]
Description=elasticsearch
After=network.target
[Service]
Type=simple
LimitMEMLOCK=infinity
WorkingDirectory=/elastic/elasticsearch/
ExecStart=/elastic/elasticsearch/bin/elasticsearch
ExecReload=/bin/kill -HUP $MAINPID #重载时执行的命令
LimitNOFILE=65536
User=elastic
PrivateTmp=true
[Install]
WantedBy=multi-user.target
EOF
systemctl enable elasticsearch
systemctl restart elasticsearch
systemctl status elasticsearch
12.生成用户和密码
//自动随机生成密码，并输出到控制台
/elastic/elasticsearch/bin/elasticsearch-setup-passwords auto
//生成的密码如下：
Changed password for user apm_system
PASSWORD apm_system = gmxadlfgV2KLdgW2uCmi
Changed password for user kibana_system
PASSWORD kibana_system = 0FmCua51aWuiuuS30Pbw
Changed password for user kibana
PASSWORD kibana = 0FmCua51aWuiuuS30Pbw
Changed password for user logstash_system
PASSWORD logstash_system = UaMTgwThhCkpwA9piaf5
Changed password for user beats_system
PASSWORD beats_system = 7SxJtzLeCfE53n8It3Qk
Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = dgkVtG5SYSmj8AN7y8PQ
Changed password for user elastic
PASSWORD elastic = Jmrb6idz2EPOlajZpIvD
13.安装kibana
mkdir /elastic/kibana
curl -O https://artifacts.elastic.co/downloads/kibana/kibana-7.15.1-linux-x86_64.tar.gz
tar -xzf kibana-7.15.1-linux-x86_64.tar.gz -C /elastic/
mv /elastic/kibana-7.15.1-linux-x86_64/ /elastic/kibana
cd /elastic/kibana/
chown -R elastic:elastic /elastic/kibana
14.修改配置文件
修改kinaba 配置文件
/elastic/kibana/config/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
server.name: "es-node1"
elasticsearch.hosts: ["http://10.0.0.4:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"
elasticsearch.username: "elastic"
elasticsearch.password: "kibana_passwd"
#如果启动用https访问
#这个是访问elk需要的，不是指kibana开启ssl
elasticsearch.ssl.certificateAuthorities: $KBN_PATH_CONF/elasticsearch-ca.pem
#如果kibana开启ssl,需要另外设置
15.设置开机启动
cat > /lib/systemd/system/kibana.service <<EOF
[Unit]
Description=kibana
After=network.target
[Service]
Type=simple
WorkingDirectory=/elastic/kibana/
ExecStart=/elastic/kibana/bin/kibana
ExecReload=/bin/kill -HUP $MAINPID #重载时执行的命令
LimitNOFILE=65536
User=elastic
PrivateTmp=true
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl start kibana
systemctl status kibana
systemctl enable kibana
以上是关于ELK集群搭建简略记录的主要内容，如果未能解决你的问题，请参考以下文章