filebeat 获取nginx日志 发送给ElasticSearch

Posted 2020-10-13

目的：通过filebeat获取nginx日志，发送给ElasticSearch，filebeat可以解析json格式日志，所以设置nginx日志为json格式。

1、配置nginx配置文件

    

    log_format jsonTest ‘{"@timestamp":"$time_iso8601",‘
                  ‘"host":"$server_addr",‘
                  ‘"service":"nginxTest",‘
                  ‘"trace":"$upstream_http_ctx_transaction_id",‘
                  ‘"log":"log",‘
                  ‘"clientip":"$remote_addr",‘
                  ‘"remote_user":"$remote_user",‘
                  ‘"request":"$request",‘
                  ‘"http_user_agent":"$http_user_agent",‘
                  ‘"size":$body_bytes_sent,‘
                  ‘"responsetime":$request_time,‘
                  ‘"upstreamtime":"$upstream_response_time",‘
                  ‘"upstreamhost":"$upstream_addr",‘
                  ‘"http_host":"$host",‘
                  ‘"url":"$uri",‘
                  ‘"domain":"$host",‘
                  ‘"xff":"$http_x_forwarded_for",‘
                  ‘"referer":"$http_referer",‘
                  ‘"status":"$status"}‘;
     access_log /var/log/nginx/access.log jsonTest;

定义jsonTest的json格式，其中trace是页面response headers的值 ctx-transaction-id，通过upstream_http_ctx_transaction_id可以获取头文件属性。

将日志输出到/var/log/nginx/access.log

2、配置filebeat配置文件

filebeat.prospectors:
- type: log
  paths:
   - ‘/root/front/logs/*.log‘
  json.message_key: log
  json.keys_under_root: true

output.elasticsearch:
  hosts: ["*.*.*.*:9200"]

将日志传输给ElasticSearch