ELK logstash 处理MySQL慢查询日志(初步)

Posted lzfj

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了ELK logstash 处理MySQL慢查询日志(初步)相关的知识,希望对你有一定的参考价值。

写在前面:在做ELK logstash 处理mysql慢查询日志的时候出现的问题:
1、测试数据库没有慢日志,所以没有日志信息,导致 IP:9200/_plugin/head/界面异常(忽然出现日志数据,删除索引后就消失了)
2、处理日志脚本问题
3、目前单节点
 
配置脚本文件/usr/local/logstash-2.3.0/config/slowlog.conf【详细脚本文件见最后】
 
output {
  elasticsearch {
    hosts => "115.28.3.150:9200"
    index => "mysql-slowlog"
    workers => 1
    flush_size => 20000
    idle_flush_time => 10
    template_overwrite => true
  }
}
在outpout定义Elasticsearch的IP与端口,以及索引名称
[root@iZ28o76f668Z config]# ../bin/logstash agent -f slowlog.conf
Settings: Default pipeline workers: 1
Pipeline main started
在http://115.28.3.150:9200/_plugin/head/页面上刷新:

-------------------------------------------------------------------------------------------------------------------------------

cat /usr/local/logstash-2.3.0/config/slowlog.conf

input {

file {
type => "mysql-slow"
path => "/mnt/data/mysql/mysql-slow.log"
start_position => "beginning"
codec => multiline {
pattern => "^# Time:"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => { "message" => "SELECT SLEEP" }
add_tag => [ "sleep_drop" ]
tag_on_failure => []
}
if "sleep_drop" in [tags] {
drop {}
}
grok {
match => [ "message", "(?m)^# Time:.*\\s+# User@Host: %{USER:user}\\[[^\\]]+\\] @ (?:(?<clienthost>\\S*) )?\\[(?:%{IP:clientip})?\\]\\s*Id: %{NUMBER:id:int}\\s+# Query_time: %{NUMBER:query_time:float}\\s+Lock_time: %{NUMBER:lock_time:float}\\s+Rows_sent: %{NUMBER:rows_sent:int}\\s+Rows_examined: %{NUMBER:rows_examined:int}\\s*(?:use %{DATA:database};\\s*)?SET timestamp=%{NUMBER:timestamp};\\s*(?<query>(?<action>\\w+)\\s+.*)$" ]
}
date {
match => [ "timestamp", "UNIX" ]
remove_field => [ "timestamp" ]
}
}
output {
elasticsearch {
hosts => "192.168.98.163:9200"
index => "mysql-slowlog"
workers => 1
flush_size => 20000
idle_flush_time => 10
template_overwrite => true
}
}

以上是关于ELK logstash 处理MySQL慢查询日志(初步)的主要内容,如果未能解决你的问题,请参考以下文章

LOGSTASH+ELASTICSEARCH 处理 MYSQL 慢查询日志

ELK采集MySQL慢日志实现

ELK采集MySQL慢日志实现

logstash解析mysql慢日志

Logstash分析MySQL慢查询日志

基于ELK搭建MySQL日志平台的要点和常见错误