filebeat 6.4.3 采集 nginx日志

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了filebeat 6.4.3 采集 nginx日志相关的知识,希望对你有一定的参考价值。

filebeat 6.4.3 采集 nginx日志

nginx 日志格式规范:

将日志配置组合添加至nginx主配置文件;

     log_format  eslog  ‘$remote_addr - $remote_user [$time_local] "$request" ‘
                      ‘$status $body_bytes_sent "$http_referer" ‘
                      ‘"$http_user_agent" "$http_x_forwarded_for"‘;

ES集群安装插件

集群收集nginx日志需要安装 ingest-user-agent ,ingest-geoip ,安装完成之后需要重启es 服务,否则数据无法正常录入es集群。

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
/usr/share/elasticsearch/bin/elasticsearch-plugin list

升级java至1.8

yum install java-1.8 -y

安装filebeat

导入key

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

创建repo 文件

more lostash.repo

[elastic-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装

yum install filebeat -y

修改filebeat主配置文件

more /etc/filebeat/filebeat.yml

filebeat.config:
prospectors:
path: ${path.config}/prospectors.d/.yml
reload.enabled: false
modules:
path: /etc/filebeat/modules.d/
.yml
reload.enabled: false

output.elasticsearch:
hosts: [‘10.2.3.30:9200‘]

setup.kibana:
host: "10.2.3.13:5601"

配置filebeat nginx 模块

Access 日志:/data/nginxlog/eslog/es-access.*
Error 日志:/data/nginxlog/eslog/es-error.*
more /etc/filebeat/modules.d/nginx.yml 
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/nginxlog/eslog/es-access.*"]
  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/data/nginxlog/eslog/es-error.*"]

添加启动

systemctl enable filebeat

开启nginx 模块

cd /etc/filebeat

filebeat modules enable nginx

filebeat modules list

初始化环境

此操作会自动导入filebeat模板和nginx dashboard 到es 集群:

Set up the initial environment:
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations
./filebeat setup -e

启动服务

systemctl start filebeat

启动服务报错

内容如下:

Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern:
  error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: returned 400 to import file: <nil>. Response: {"statusCode":400,"error":"Bad Request","message":"Request Timeout after 30000ms"}

2018-10-31T16:35:45.659+0900    INFO    kibana/client.go:113    Kibana url: http://10.2.3.13:5601
2018-10-31T16:37:15.664+0900    ERROR   instance/beat.go:743    Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/filebeat/kibana: Failed to import index-pattern: Failed to load directory /usr/share/filebeat/kibana/6/index-pattern:
  error loading /usr/share/filebeat/kibana/6/index-pattern/filebeat.json: fail to execute the HTTP POST request: Post http://10.2.3.13:5601/api/kibana/dashboards/import?force=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers). Response: 

处理办法:

登陆kibana 进入Dev tools 删除

删除旧模板
    #DELETE _template/filebeat-6.X.X 
删除旧数据
    #DELETE filebeat-6.4.2-*
重新导入模板:
cd /etc/filebeat/
filebeat setup
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded machine learning job configurations
重启服务:
systemctl restart filebeat

Dashboard 效果:

技术分享图片

以上是关于filebeat 6.4.3 采集 nginx日志的主要内容,如果未能解决你的问题,请参考以下文章

filebeat采集nginx日志,业务日志,阿里云sms,slb日志

filebeat采集nginx日志,业务日志,阿里云sms,slb日志

ELK(Elasticsearch+Filebeat+Kibana) 轻量级采集分析Nginx日志

ELK(Elasticsearch+Filebeat+Kibana) 轻量级采集分析Nginx日志

Filebeat同时采集多个日志

Filebeat 与Kibana仪表板(二十一)