《果然新鲜》电商项目（48）- Docker下Kibana与Logstash的ES集群配置

Posted 2022-02-27 IT-老牛

文章目录

• 1. kibana配置ES集群
• • 1.1.新建宿主主机的挂载配置文件:
  • 1.2.以挂载配置文件的方式启动kibana
• 1.3.浏览器访问
• 2. logstash配置ES集群
• • 2.1.新建宿主主机的挂载配置文件:
  • 2.2 以挂载配置文件的方式启动logstash
• 3. 测试
• 4. 总结

1. kibana配置ES集群

1.1.新建宿主主机的挂载配置文件:

cd /usr/local
mkdir -p kibana/config
cd kibana
cd config
vi kibana.yml

配置内容如下：

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.10.128:9201","http://192.168.10.128:9200"]

字段解析以及填坑方法：

1.2.以挂载配置文件的方式启动kibana

docker pull kibana:7.6.1

docker run \\
-d -p 5601:5601 \\
-v /usr/local/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \\
--name kibana-cluster kibana:7.6.1

1.3.浏览器访问

访问http://192.168.10.128:5601/app/kibana#/dev_tools/console，可以看到启动成功：

2. logstash配置ES集群

2.1.新建宿主主机的挂载配置文件:

cd /usr/local
mkdir -p logstash/config
cd logstash
mkdir lib
cd config
vi mysql.conf
vi mysql1.conf
vi pipelines.yml

上传jar包到/user/local/logstash/lib目录下：

配置：mysql.conf、mysql1.conf、pipelines.yml

1. mysql.conf内容（注意es集群配置）

input 
  jdbc 
    jdbc_driver_library => "/usr/share/logstash/lib/mysql-connector-java-5.1.17-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://172.81.235.217:3306/guoranxinxian-goods"
    jdbc_user => "root"
    jdbc_password => "bruce_1234"
    schedule => "* * * * *"
    statement => "SELECT * FROM product WHERE UPDATED_TIME > :sql_last_value"
    use_column_value => true
    tracking_column_type => "timestamp"
    tracking_column => "updated_time"
    last_run_metadata_path => "syncpoint_product_table"
  



output 
    elasticsearch 
        hosts => ["192.168.10.128:9200","192.168.10.128:9201"]
        index => "product"
        document_id => "%id"
        document_type => "product"
    
    stdout 
        codec => json_lines
    

2. mysql1.conf内容（注意es集群配置）

input 
  jdbc 
    jdbc_driver_library => "/usr/share/logstash/lib/mysql-connector-java-5.1.17-bin.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://172.81.235.217:3306/guoranxinxian-goods"
    jdbc_user => "root"
    jdbc_password => "bruce_1234"
    schedule => "* * * * *"
    statement => "SELECT * FROM category WHERE UPDATED_TIME > :sql_last_value"
    use_column_value => true
    tracking_column_type => "timestamp"
    tracking_column => "updated_time"
    last_run_metadata_path => "syncpoint_category_table"
  



output 
    elasticsearch 
        hosts => ["192.168.10.128:9200","192.168.10.128:9201"]
        index => "category"
        document_id => "%id"
        document_type => "category"
    
    stdout 
        codec => json_lines
    

3. pipelines.yml内容：

- pipeline.id: product_table
  path.config: "/usr/share/logstash/config/mysql.conf"
- pipeline.id: category_table
  path.config: "/usr/share/logstash/config/mysql1.conf"

2.2 以挂载配置文件的方式启动logstash

mysql.conf、mysql1.conf、pipelines.yml

docker run \\
-v /usr/local/logstash/lib/mysql-connector-java-5.1.17-bin.jar:/usr/share/logstash/lib/mysql-connector-java-5.1.17-bin.jar \\
-v /usr/local/logstash/config/mysql.conf:/usr/share/logstash/config/mysql.conf \\
-v /usr/local/logstash/config/mysql1.conf:/usr/share/logstash/config/mysql1.conf \\
-v /usr/local/logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml \\
-d --name logstash-cluster logstash:7.6.1

3. 测试

使用kibana查询商品product是否插入ES集群，下图可以看出插入成功:

使用kibana查询商品目录category是否插入ES集群，下图可以看出插入成功:

4. 总结

本文主要讲解Kibana和Logstash配置集群，主要内容如下：