logstash同步MySQL数据到Elasticsearch

Posted 2021-03-10 王晓天

1. 从Spring Boot下载项目

https://spring.io/guides/gs/accessing-data-mysql/

 

2. 配置Spring Boot连接MySQL(位置:resources/application.properties)

spring.jpa.hibernate.ddl-auto=update
spring.datasource.urlcurl \'localhost:8080/demo/all\'=jdbc:mysql://localhost:3306/db_example?useUnicode=true&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC
spring.datasource.username=XXX
spring.datasource.password=XXX

注意事项：jdbc_connection_string字符串一定添加

?useUnicode=true&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC

 

3. 配置logstash

input {
jdbc {
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/db_example?useUnicode=true&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC"
jdbc_user => "XXX"
jdbc_password => "XXX"
use_column_value => true
tracking_column => "last_updated"
tracking_column_type => "numeric"
record_last_run => true
last_run_metadata_path => "tmp.txt"
statement => "SELECT * FROM user where last_updated >:sql_last_value;"
schedule => " * * * * * *"
}
}
output {
elasticsearch {
document_id => "%{id}"
document_type => "_doc"
index => "users"
hosts => ["http://localhost:9200"]
}
stdout{
codec => rubydebug
}
}

 

注意事项:
1)jdbc_connection_string字符串一定添加

?useUnicode=true&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC

2)logstash配置文件删除注释

3)配置文件路径:logstash-7.3.2\\bin\\mysql.conf 必须是conf文件

4. 将mysql-connector-java-8.0.19.jar复制到logstash-7.3.2\\logstash-core\\lib\\jars
注意：mysql-connector-java-8.0.19.jar是在Spring Boot Maven项目配置MySQL后自动下载的

 

5. 启动Spring Boot，Elasticsearch，logstash，Kibana（ELK套件版本要一致）

ref:
https://www.cnblogs.com/jj81/p/9456448.html
https://www.cnblogs.com/jeffen/p/6288142.html