ELFK部署

Posted 还行少年

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了ELFK部署相关的知识,希望对你有一定的参考价值。

这里写自定义目录标题

官方下载地址
https://www.elastic.co/cn/downloads/past-releases#elasticsearch
https://www.elastic.co/cn/downloads/past-releases#kibana
https://www.elastic.co/cn/downloads/past-releases#logstash
https://www.elastic.co/cn/downloads/past-releases#filebeat

1、Elasticsearch部署

# a.创建用户
groupadd es
useradd es -g es

# b.下载安装包 (LINUX下 elasticsearch 无法使用 root 用户运行,需要创建新的用户,并且把elasticsearch 下所有的文件更改所属用户)
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.4.3-linux-x86_64.tar.gz
tar xf elasticsearch-8.4.3-linux-x86_64.tar.gz -C /data
chown -R es:es /data/elasticsearch-8.4.3/

# c.修改配置文件
cat  <<EOF  > /data/elasticsearch-8.4.3/config/elasticsearch.yml
cluster.name: my-es
node.name: es
path.data: /data/elasticsearch-8.4.3/data
path.logs: /data/elasticsearch-8.4.3/logs
network.host: 192.168.30.10
http.port: 9200
discovery.seed_hosts: ["192.168.30.10:9300"]
cluster.initial_master_nodes: ["es"]
index.store.type: niofs
bootstrap.memory_lock: true 
indices.requests.cache.size: 5%
indices.queries.cache.size: 10%
## 开启配置密码认证
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: elastic-certificates.p12
  truststore.path: elastic-certificates.p12
EOF

# d.修改JVM,配置文件,此处设置ES所使用的内存,需小于系统空闲内存,否则无法启动。一般设置为服务器剩余内存的一半,且不能超过32G。最小和最大必须设置一样,避免GC(垃圾回收)
sed -i 's/^## -Xms4g/-Xms2g/g ' /data/elasticsearch-8.4.3/config/jvm.options
sed -i 's/^## -Xmx4g/-Xmx2g/g ' /data/elasticsearch-8.4.3/config/jvm.options


# e.切换用户生成证书文件(设置密码用,默认回车即可)
su es
cd /data/elasticsearch-8.4.3/bin/
./elasticsearch-certutil ca
./elasticsearch-certutil cert --ca elastic-stack-ca.p12
mv /data/elasticsearch-8.4.3/elastic-certificates.p12 ../config

# f.启动 elasticsearch并检查进程和端口是否正常
./elasticsearch -d
ps -ef | grep elastic 
netstat -natp | grep 9200

# g.手动设置密码
./elasticsearch-reset-password -u elastic -i
./elasticsearch-reset-password -u kibana -i

# h.验证密码

2、logstash与filebeat部署

# a.下载安装包
wget https://artifacts.elastic.co/downloads/logstash/logstash-8.4.3-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.3-linux-x86_64.tar.gz
tar xf logstash-8.4.3-linux-x86_64.tar.gz  -C /data
tar xf filebeat-8.4.3-linux-x86_64.tar.gz 
mv filebeat-8.4.3-linux-x86_64 /data/filebeat-8.4.3

# b.安装java环境
yum -y install java-11-openjdk
# c1.filebeat直接发送数据到es(filebeat同时只支持一个output)
cat /data/filebeat-8.4.3/test.yml

filebeat.inputs:
- type: log
  id: nginx
  enabled: true
  paths:
    - /var/log/nginx/access.log

output.elasticsearch:
  hosts: ["192.168.30.10:9200"]
  username: "elastic"
  password: "123456"
  index: "nginx-%+yyyy.MM.dd"

setup.ilm.enabled: false 
setup.template.enabled: false 
setup.template.overwrite: true
# c2.filebeat发送数据到logstash
filebeat.inputs:
- type: log
  id: nginx
  enabled: true
  paths:
    - /var/log/nginx/access.log
    
output.logstash:
  enabled: true
  hosts: ["192.168.30.30:5044"]
# c3.启动filebeat
cd /data/filebeat-8.4.3/
nohup ./filebeat -e -c test.yml & 
# d.logstah收集过滤数据
cat /data/logstash-8.4.3/config/test.yml

input 
#从filebeat获取nginx日志
  beats 
    host => "192.168.30.30"
    port => 5044
    type => "nginx-filebeat"
   
#从本地获取日志  
  file 
    path => "/var/log/nginx/access.log"
    type => "nginx-logstash"
    start_position => "beginning"
  


output 
#根据不同的type来命名索引
  if [type] == "nginx-filebeat" 
  elasticsearch 
    hosts => ["http://192.168.30.10:9200"]
    index => "nginx-filebeat-%+YYYY.MM.dd"
    user => "elastic"
    password => "123456"
  
  
  if [type] == "nginx-logstash" 
  elasticsearch 
    hosts => ["http://192.168.30.10:9200"]
    index => "nginx-logstash-%+YYYY.MM.dd"
    user => "elastic"
    password => "123456"
  
  


# 启动logstash
cd /data/logstash-8.4.3/bin/
nohup ./logstash -f /data/logstash-8.4.3/config/test.yml  &
# e.在es中查看索引

3、Kibana部署

# a.创建用户
groupadd kibana
useradd kibana -g kibana

# b.下载安装包
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.4.3-linux-x86_64.tar.gz
tar xf kibana-8.4.3-linux-x86_64.tar.gz -C /data
chown -R kibana /data/kibana-8.4.3/

# c.修改配置文件
grep -Ev "^$|^#" /data/kibana-8.4.3/config/kibana.yml 
## 过滤查看修改的配置如下:
server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "http://192.168.30.20:5601"
server.name: "kibana"
elasticsearch.hosts: ["http://192.168.30.10:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "123456"

# d.启动kibana
su kibana
cd /data/kibana-8.4.3/bin/
nohup ./kibana &

# e.查看进程与端口
netstat -antp | grep 5601
ps -ef | grep kibana

# f.登录
ip:port
es的账号密码

# g.关联es索引

# h.查看kibana视图

以上是关于ELFK部署的主要内容,如果未能解决你的问题,请参考以下文章

ELFK Filebeat+ELK 部署 zookeeper集群+kafka集群 部署

yum搭建ELFK日志采集系统

在ELFK架构中加入kafka

elfk 搭建系列 -- logstash 的搭建

ELK+kafka+filebeat搭建生产ELFK集群 --markdown 语法

linux 搭建ELFK6.8.0集群