logstash 和 filebeat 连接有用户名密码的elasticsearch
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了logstash 和 filebeat 连接有用户名密码的elasticsearch相关的知识,希望对你有一定的参考价值。
参考技术A[图片上传失败...(image-61c09e-1619429119566)]
文件夹创建整体架构图
参考资料
vi ~/.bash_profile
export JAVA_HOME=/usr/local/apps/jdk1.8.0_261
export PATH= JAVA_HOME/bin
export CLASSPATH=.: JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
source ~/.bash_profile
可以访问jdk
chmod -r 755 jdk1.8.0_261
https://blog.csdn.net/ubuntutouch/article/details/100675502
[图片上传失败...(image-af15e0-1619429119566)]
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n438" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit;">mkdir /usr/local/apps/logstash
chmod 775 /usr/local/apps/logstash
mkdir /usr/local/apps/filebeat
chmod 775 /usr/local/apps/filebeat
mkdir /srv/logstash/log
mkdir /srv/logstash/data
chmod -R 775 /srv/logstash
mkdir /srv/filebeat/log
mkdir /srv/filebeat/data
chmod -R 775 /srv/filebeat
cd /usr/local/apps/logstash</pre>
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.11.2-linux-x86_64.tar.gz
解压程序 tar -xvf logstash-7.11.2-linux-x86_64.tar.gz
在config下创建 mylogstash-filebeat.conf文件
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n58" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit;">input
beats
port => 5044
output
elasticsearch
hosts => ["http://172.20.3.12:9200","http://172.20.3.35:9200","http://172.20.3.36:9200"]
user => "elastic"
password => "gVZNqhqu636wm13h8npd"
index => "logstash"
</pre>
vi config/logstash.yml
path.data: /srv/data/logstash
nohup ./logstash -f ../config/mylogstash-kafka.conf &
ps -ef | grep logstash
tail -100f nohup.out
:error=>"Got response code \'401\' contacting Elasticsearch at UR
可能是你没有配置用户名密码
curl -XGET \'localhost:9600/?pretty\'
cd /usr/local/apps/filebeat
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.1-linux-x86_64.tar.gz
tar xzvf filebeat-7.3.1-linux-x86_64.tar.gz
cd ilebeat-7.3.1-linux-x86_64
mv * ../
filebeat 日志目录配置
抓取扫描文件配置
vi /usr/local/apps/filebeat/filebeat.yml
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n340" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit;">打开以下内容
打开使能
#抓取日志的配置
enabled: true
paths:
• - /srv/filebeat/log/*.log
• - /usr/local/apps/filebeat/test.log
#logstash的配置
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
#日志路径配置
logging.to_files: true
logging.files:
path: /srv/filebeat/log
name: filebeat
keepfiles: 7
permissions: 0644
</pre>
权限控制
<pre mdtype="fences" cid="n108" lang="" class="md-fences md-end-block ty-contain-cm modeLoaded" spellcheck="false" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit;">把文件加权限赋予rd账户
chown -R rd:rd /usr/local/apps/filebeat
后面的切换到rd账户执行
su rd
chmod go-w /usr/local/apps/filebeat/filebeat.yml
touch /usr/local/apps/filebeat/test.log
echo "asdfasdfasdf" >>/usr/local/apps/filebeat/test.log
nohup ./filebeat &
tail -fn 200 nohup.out</pre>
./filebeat test output 测试运行状态
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n387" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit;">rd@localhost filebeat]$ ./filebeat test output
logstash: localhost:5044...
connection...
• parse host... OK
• dns lookup... OK
• addresses: ::1, 127.0.0.1
• dial up... OK
TLS... WARN secure connection disabled
talk to server... OK</pre>
说明是正常的
查询es
curl -XGET --user elastic:gVZNqhqu636wm13h8npd \' http://172.20.3.35:9200/logstash/_search \'
<pre spellcheck="false" class="md-fences md-end-block ty-contain-cm modeLoaded" lang="" cid="n415" mdtype="fences" style="box-sizing: border-box; overflow: visible; font-family: var(--monospace); font-size: 0.9em; display: block; break-inside: avoid; text-align: left; white-space: normal; background-image: inherit; background-position: inherit; background-size: inherit; background-repeat: inherit; background-attachment: inherit; background-origin: inherit; background-clip: inherit; background-color: rgb(248, 248, 248); position: relative !important; border: 1px solid rgb(231, 234, 237); border-radius: 3px; padding: 8px 4px 6px; margin-bottom: 15px; margin-top: 15px; width: inherit;">"_index":"logstash","_type":"_doc","_id":"HtlkDXkBEawMgE7wuu-Z","_score":1.0,"_source":"offset":135,"source":"/usr/local/apps/filebeat/test.log","prospector":"type":"log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.094Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"yjhkDXkBo6lKLCg_uhM3","_score":1.0,"_source":"offset":151,"source":"/usr/local/apps/filebeat/test.log","prospector":"type":"log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.094Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"yzhkDXkBo6lKLCg_uhM3","_score":1.0,"_source":"offset":503,"source":"/usr/local/apps/filebeat/test.log","prospector":"type":"log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.096Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"yDhkDXkBo6lKLCg_uhM0","_score":1.0,"_source":"offset":711,"source":"/usr/local/apps/filebeat/test.log","prospector":"type":"log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.097Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"wDhkDXkBo6lKLCg_uhM0","_score":1.0,"_source":"offset":169,"source":"/srv/filebeat/log/a.log","prospector":"type":"log","message":"asdfasdfasdf","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.093Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"wThkDXkBo6lKLCg_uhM0","_score":1.0,"_source":"offset":279,"source":"/usr/local/apps/filebeat/test.log","prospector":"type":"log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.095Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"HdlkDXkBEawMgE7wuu-V","_score":1.0,"_source":"offset":423,"source":"/usr/local/apps/filebeat/test.log","prospector":"type":"log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.096Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"wjhkDXkBo6lKLCg_uhM0","_score":1.0,"_source":"offset":743,"prospector":"type":"log","source":"/usr/local/apps/filebeat/test.log","message":"123123123123123","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.097Z","tags":["beats_input_codec_plain_applied"],"_index":"logstash","_type":"_doc","_id":"DdlkDXkBEawMgE7wuu-O","_score":1.0,"_source":"offset":39,"prospector":"type":"log","source":"/usr/local/apps/filebeat/test.log","message":"asdfasdfasdf","@version":"1","beat":"version":"6.2.4","name":"localhost.localdomain","hostname":"localhost.localdomain","@timestamp":"2021-04-26T08:56:14.094Z","tags":["beats_input_codec_plain_applied"]]</pre>
参考:
https://www.cnblogs.com/xiaobaozi-95/p/9214307.html
https://blog.csdn.net/zjcjava/article/details/99258682
https://www.jianshu.com/p/4aa3a8b70bfa
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html
配置里抓取的日志 但是感觉filebeat 没有抓取日志
模拟产生日志
http://172.20.3.35:9200/activecard-performance-2021.04.26/_search
http://172.20.3.35:9200/activecard-performance-2021.04.26/_search
查询日志
不行 那个用户只能查询
elasticsearch 多个用户权限问题
ELK 堆栈中的 Logstash 和 filebeat
【中文标题】ELK 堆栈中的 Logstash 和 filebeat【英文标题】:Logstash and filebeat in the ELK stack 【发布时间】:2019-08-16 20:14:04 【问题描述】:我们正在服务器上设置 elasticsearch、kibana、logstash 和 filebeat,以分析来自许多应用程序的日志文件。由于原因*,每个应用程序日志文件最终都位于 ELK 服务器上的单独目录中。我们有大约 20 个日志文件。
-
据我了解,我们可以为每个运行一个 logstash 管道配置文件
应用程序日志文件。这将是一个正在运行的 logstash 实例
有 20 条并行管道,每条管道都需要自己的
击败港口。请确认这是否正确?
我们可以运行一个 filebeat 实例,还是每个都需要一个
管道/日志文件?
此架构是否可行,或者您是否发现任何主要缺点?
谢谢!
*不同的供应商负责不同的应用程序,他们跨多个不同的操作系统运行,其中许多不会或不能安装像 filebeats 这样的东西。
【问题讨论】:
filebeat 应该在有应用程序日志的同一台服务器上,并且您可以将一个 filebeat 配置为读取多个文件 感谢您的回复,但就我而言,无法在运行应用程序的服务器上安装 filebeat。相反,文件将从 ELK 服务器读取。您能否分享有关文件节拍和许多日志的文档的链接? 检查我的答案 【参考方案1】:我们不建议从网络卷中读取日志文件。每当 可能,在主机上安装 Filebeat 并发送日志文件 直接从那里。从网络卷中读取文件(尤其是在 Windows)可能会产生意想不到的副作用。例如,更改的文件 标识符可能会导致 Filebeat 从头开始读取日志文件 再次。
Reference
我们始终建议在远程服务器上安装 Filebeat。使用 不支持共享文件夹。典型的设置是你有一个 Logstash + Elasticsearch + Kibana 在一个中心位置(一个或多个 服务器)和 Filebeat 安装在您所在的远程计算机上 正在收集数据。
Reference
对于一个运行的 filebeat 实例,您可以通过定义多个输入部分来将不同的配置设置应用于不同的文件,如下例,check here for more
filebeat.inputs:
- type: log
enabled: true
paths:
- 'C:\App01_Logs\log.txt'
tags: ["App01"]
fields:
app_name: App01
- type: log
enabled: true
paths:
- 'C:\App02_Logs\log.txt'
tags: ["App02"]
fields:
app_name: App02
- type: log
enabled: true
paths:
- 'C:\App03_Logs\log.txt'
tags: ["App03"]
fields:
app_name: App03
您可以在过滤器中使用带有 if 语句的 logstash 管道
filter
if [fields][app_name] == "App01"
grok
else if [fields][app_name] == "App02"
grok
else
grok
条件也可以是 if "App02" in [tags]
或 if [source]=="C:\App01_Logs\log.txt"
,因为我们从 filebeat 发送
【讨论】:
感谢您的反馈和链接。 1.如果我不允许在目标机器上安装 filebeat,你建议什么替代方案? 2. 由于我正在阅读看起来非常不同的日志文件,我的印象是我需要将它们发送到不同的 logstash 管道以进行非常不同的 grok 解析,因此它需要不同的 filebeat.outputs 但就我而言这是不允许的理解。我理解错了吗? @user1329339 对于第 1 点我现在没有建议,对于每个输入部分的第 2 点,您可以拥有例如tags: ["App01"]
并在 logstash 中使用 if 语句来检查标签并做不同的 grok解析
@user1329339 我已经更新了我的答案以获取更多详细信息
再次感谢您的反馈。不幸的是,我无法在不同的服务器上安装 filebeats。是否可以使用 logstash 文件作为输入而不是 beats 并且对于不同的管道有不同的路径?
@user1329339 检查此链接以获取logstash 文件elastic.co/guide/en/logstash/current/…,通常您将如何访问当前不同服务器的日志?以上是关于logstash 和 filebeat 连接有用户名密码的elasticsearch的主要内容,如果未能解决你的问题,请参考以下文章