docker-compose ELK查看docker及容器的日志

Posted weschen

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了docker-compose ELK查看docker及容器的日志相关的知识,希望对你有一定的参考价值。

我目前所在公司开发团队比较小,为集团下面的工厂开发了一套小的系统,跑在一台CentOS服务器上,服务器搭建了docker环境,安装了docker-compose,但在日志处理方面,暂时没有一个好的方法能够收集完全的日志,只能依赖进入至服务器后,以docker logs containerID的方法来进入查看,非常不方便,之前也有关注ELK的技术,但一直在开发系统功能,全力实现,今天得空,重新想起了ELK查看日志的任务。

 

项目文件夹

技术图片

其中docker-compose.yml

version: ‘3‘

services:
  filebeat:
    hostname: filebeat
    image: weschen/filebeat
    build:
      context: filebeat
      dockerfile: Dockerfile
    volumes:
      # needed to access all docker logs (read only) :
     - "/var/lib/docker/containers:/usr/share/dockerlogs/data:ro"
      # needed to access additional informations about containers
     - "/var/run/docker.sock:/var/run/docker.sock"
    links:
       - logstash
  kibana:
    image: docker.elastic.co/kibana/kibana:6.5.2
    environment:
      - "LOGGING_QUIET=true"
    links:
      - elasticsearch
    ports:
      - 5601:5601
  logstash: 
    hostname: logstash 
    image: weschen/logstash
    build:
      context: logstash
      dockerfile: Dockerfile
    ports:
      - 5044:5044
    environment:
      LOG_LEVEL: error
    links:
      - elasticsearch
  elasticsearch:
    hostname: elasticsearch
    image: weschen/elasticsearch
    build:
      context: elasticsearch
      dockerfile: Dockerfile
    environment:
      - cluster.name=docker-elk-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200

 

 

1.Elasticsearch

文件elasticsearch/Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:6.5.2
COPY --chown=elasticsearch:elasticsearch elasticsearch.yml /usr/share/elasticsearch/config/

CMD ["elasticsearch", "-Elogger.level=INFO"]

 

文件elasticsearch/elasticsearch.yml

cluster.name: $cluster.name
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

 

 

2.Logstash

文件logstash/Dockerfile

FROM docker.elastic.co/logstash/logstash:6.5.2

RUN rm -f /usr/share/logstash/pipeline/logstash.conf
COPY pipeline /usr/share/logstash/pipeline/

 

文件logstash/pipeline/logstash.conf

input  
    beats 
        port => 5044
        host => "0.0.0.0"
      
 

output  
    elasticsearch  
        hosts => elasticsearch
        manage_template => false
        index => "%[@metadata][beat]-%[@metadata][version]-%+YYYY.MM.dd"
     
    
    stdout  codec => rubydebug 

 

 

 

3.Filebeat

文件filebeat/Dockerfile

FROM docker.elastic.co/beats/filebeat:6.5.2

# Copy our custom configuration file
COPY filebeat.yml /usr/share/filebeat/filebeat.yml

USER root
# Create a directory to map volume with all docker log files
RUN mkdir /usr/share/filebeat/dockerlogs
RUN chown -R root /usr/share/filebeat/
RUN chmod -R go-w /usr/share/filebeat/

 

文件filebeat/filebeat.yml

filebeat.inputs:
- type: docker
  combine_partial: true
  containers:
    path: "/usr/share/dockerlogs/data"
    stream: "stdout"
    ids:
      - "*"
  exclude_files: [‘\\.gz$‘]
  ignore_older: 10m

processors:
  # decode the log field (sub JSON document) if JSON encoded, then maps it‘s fields to elasticsearch fields
- decode_json_fields:
    fields: ["log", "message"]
    target: ""
    # overwrite existing target elasticsearch fields while decoding json fields    
    overwrite_keys: true
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

filebeat.config.modules:
  path: $path.config/modules.d/*.yml
  reload.enabled: false

# setup filebeat to send output to logstash
output.logstash:
  hosts: ["logstash"]

# Write Filebeat own logs only to file to avoid catching them with itself in docker log files
logging.level: error
logging.to_files: false
logging.to_syslog: false
loggins.metrice.enabled: false
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
ssl.verification_mode: none

 

 

 

使用docker-compose up -d跑起来

技术图片

 

 

在浏览器打开[Host-IP]:9200,能够打开以下界面,说明elasticsearch服务已经起来了

技术图片

 

 

再在浏览器打开[Host-IP]:5601,这个是Kibana日志查看平台

技术图片

 

创建index-pattern操作如下,如果在Discover菜单中创建index-pattern时,会出现以下

技术图片

 

创建了index-pattern后,查看Logs应该可以查看到日志

技术图片

 

首页查看日志

技术图片

 

 

 

源码地址:https://github.com/ChenWes/docker-elk

以上是关于docker-compose ELK查看docker及容器的日志的主要内容,如果未能解决你的问题,请参考以下文章

分布式ELK+KAFKA日志采集 docker-compose

docker-compose部署ELK(亲测)

使用 docker-compose 为 ELK 推荐的 RAM 比率

docker-compose安装

Springboot--ELK快速搭建

基于docker-compose的 ELK5.5.1+logback 日志系统搭建