elk平台集成log4jlog4j2logback日志
Posted 奥林匹斯圣山
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了elk平台集成log4jlog4j2logback日志相关的知识,希望对你有一定的参考价值。
elk平台集成log4j、log4j2、logback日志
目前应用开发使用主流的日志框架有
log4j
、log4j2
、logback
,elk
平台集成三种日志框架,实现日志集中收集显得很有必要,接下主要分享elk集成的方法及步骤:
由于elk平台版本级别的影响,主要分两部分进行进行介绍:5.0版本elk
平台与6.0版本的elk
平台集成log4j
、log4j2
、logback
日志
5.0版本elk平台集成 log4j、log4j2、logback日志
5.0版本的
logstash
提供专门的log4j
端口(4560)用于接收并解析log4j的日志。log4j2
、logback
日志框架生成json
格式日志,logstash
通过使用TCP
Plugin
收集日志,无需做grok
的解析。
logback
配置
pom.xml
文件中添加如下内容
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>4.9</version>
</dependency>
编辑
logback.xml
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder charset="UTF-8"> <!-- encoder 可以指定字符集,对于中文输出有意义 -->
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger - %msg%n
</pattern>
</encoder>
</appender>
<appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<destination>10.72.243.131:5046</destination>
<!-- encoder is required -->
<encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" />
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"application":"myApplication"}</customFields>
</encoder>
</appender>
<root level="info"> <!-- 设置日志级别 -->
<appender-ref ref="STDOUT" />
<appender-ref ref="stash" />
</root>
log4j
配置
log4j-1.2.17.jar
、log4j-socketappender-hostname.jar
和jsonevent-layout-1.7.jar
放到/usr/local/apache-tomcat-7.0.69/webapps/应用/WEB-INF/lib
,两种配置文件将日志输送到elk平台
编辑 log4j.properties
添加以下内容(
`/usr/local/apache-tomcat-7.0.69/webapps/application/WEB-INF/classes
)
log4j.rootLogger
log4j.rootLogger=info,stdout,info,debug,error,Kibana
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=net.logstash.log4j.JSONEventLayoutV1
log4j.appender.Kibana=org.apache.log4j.net.SocketAppenderWithHostname
log4j.appender.Kibana.Port=4560
log4j.appender.Kibana.RemoteHost=10.72.243.131
log4j.appender.Kibana.layout=net.logstash.log4j.JSONEventLayoutV1
log4j.appender.Kibana.ReconnectionDelay=10000
log4j.appender.Kibana.LocationInfo=true
log4j.appender.Kibana.application=application名字
log4j.appender.Kibana.MdcKeyForHostname=Docker_Name
编辑log4j.xml
添加以下内容(
/usr/local/apache-tomcat-7.0.69/webapps/application/WEB-INF/classes
)
<appender name="logserver" class="org.apache.log4j.net.SocketAppenderWithHostname">
<param name="Port" value="4560"/>
<param name="RemoteHost" value="10.72.2413.131"/>
<param name="ReconnectionDelay" value="10000"/>
<param name="LocationInfo" value="true"/>
<param name="application" value="application名字"/>
<param name="MdcKeyForHostname" value="Docker_Name"/>
<layout class="net.logstash.log4j.JSONEventLayoutV1">
</layout>
</appender>
log4j2
配置
配置
pom.xml
文件
<!—从Spring中取消默认logback -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
</dependency>
<!—从Spring中增加log4j2 -->
<!- 开发不使用Maven和Spring的,要自己增加相关的Jar包引用—>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-log4j2</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
配置
log4j2.xml
文件
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="warn">
<Properties>
<Property name="hostName2">$${ctx:hostName}</Property>
<property name="applicationName">Demo8</property>
</Properties>
<appenders>
<console name="Console" target="SYSTEM_OUT">
<JSONLayout complete="false"></JSONLayout>
</console>
<Socket name="Logstash" host="10.72.243.131" port="5046"
protocol="TCP">
<JSONLayout complete="false" compact="true" eventEol="true"
properties="true">
</JSONLayout>
</Socket>
</appenders>
<loggers>
<root level="error">
<property name="hostName">${hostName}</property>
<property name="applicationName">"${applicationName}"</property>
<appender-ref ref="Logstash" />
<appender-ref ref="Console" />
</root>
</loggers>
</configuration>
6.0版本elk平台集成 log4j、log4j2、logback日志
elk平台升级至6.0以上的版本以后发现logstash不再支持log4j应用日志写入,另外发现应用日志直接写入logstash端口的方式会存在日志丢失的情况,鉴于以上情况,我们修改了日志应用日志收集的方式,使用新方式进行收集:应用日志输出至指定目录文件,并指定按照日志目录中文件的大小及个数进行轮询,使用filebeat读取应用日志,详见如下:
log4j
配置
pom.xml
文件引用如下jar
包
<dependency>
<groupId>net.logstash.log4j</groupId>
<artifactId>jsonevent-layout</artifactId>
<version>1.7</version>
</dependency>
配置
log4j.properties
文件
log4j.appender.RollFileToLogCenter=org.apache.log4j.RollingFileAppender
log4j.appender.RollFileToLogCenter.Threshold=DEBUG
log4j.appender.RollFileToLogCenter.File=/var/log/app/应用名称.log
log4j.appender.RollFileToLogCenter.layout=net.logstash.log4j.JSONEventLayoutV1
log4j.appender.RollFileToLogCenter.layout.UserFields=application: 应用名称
log4j.appender.RollFileToLogCenter.MaxFileSize=10MB
log4j.appender.RollFileToLogCenter.MaxBackupIndex=10
logback
配置
pom.xml
文件引用如下jar
包
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>5.1</version>
</dependency>
编辑
logback.xml
文件<?xml version="1.0" encoding="UTF-8"?> <configuration> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="net.logstash.logback.encoder.LogstashEncoder" > <customFields>{"application":"应用名称"}</customFields> </encoder> </appender> <appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="FILE_INFO"> <file>/var/log/app/application名字.log</file> <encoder class="net.logstash.logback.encoder.LogstashEncoder" > <customFields>{"application":"应用名称"}</customFields> </encoder> <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy"> <fileNamePattern>/var/log/app/application名字i.log</fileNamePattern> <minIndex>1</minIndex> <maxIndex>9</maxIndex> </rollingPolicy> <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"> <maxFileSize>2MB</maxFileSize> </triggeringPolicy> </appender> <root level="INFO"> <appender-ref ref="STDOUT" /> <appender-ref ref="FILE_INFO" /> </root> </configuration>
log4j2
配置
pom.xml
文件引用如下jar
包
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.5.3</version>
</dependency>
<dependency>
<groupId>net.logstash.log4j</groupId>
<artifactId>jsonevent-layout</artifactId>
<version>1.7</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.5.3</version>
</dependency>
编辑
log4j2.xml
文件
<Properties>
<Property name="hostName2">$${ctx:hostName}</Property>
<property name="application">应用名称</property>
</Properties>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<JsonLayout eventEol="true" complete="false" properties="true" />
</Console>
<RollingFile name="RollingFileInfo" fileName="/var/log/app/application名字.log" filePattern="/var/log/app/application名字i.log">
<JsonLayout eventEol="true" compact="true" complete="false" properties="true" />
<Policies>
<SizeBasedTriggeringPolicy size="10MB" />
</Policies>
<DefaultRolloverStrategy max="9" />
</RollingFile>
</Appenders>
<Loggers>
<Root level="INFO">
<property name="hostName">${hostName}</property>
<property name="application">${application}</property>
<AppenderRef ref="RollingFileInfo"/>
<AppenderRef ref="Console" />
</Root>
</Loggers>
</Configuration>
ELK平台配置
logstash.conf
添加如下内容:
input{
log4j {
type => "log4j"
port => 4560
}
tcp {
port => 5046
tags => ["tags"]
codec => json_lines
}
beats { port => 5044 }
}
重启
logstash
服务,查看端口监听状态:
#systemctl stop logstash
#systemctl start logstash
#lsof -i:4560
#lsof -i:5046
以上是关于elk平台集成log4jlog4j2logback日志的主要内容,如果未能解决你的问题,请参考以下文章
elasticsearch kibana logstash(ELK)的安装集成应用
原创投稿 | 一键启动 filebeat 5.1.1 集成 logstash
springboot集成elk 二:springboot + elk 采集日志