docker安装ELK
Posted dalianpai
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了docker安装ELK相关的知识，希望对你有一定的参考价值。
3个容器需要进行交互，根据docker的网络原理，docker0网桥要有，之前因为网桥不在，虽然es启动起来，但是一直ping不通，logstash也连不上es,而且存在缺陷，容器的ip可能会发生变化，后面会将用k8s去部署。
[root@topcheer ~]# ifconfig docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet6 fe80::42:ff:fe12:a43b prefixlen 64 scopeid 0x20<link> ether 02:42:00:12:a4:3b txqueuelen 0 (Ethernet) RX packets 38 bytes 3028 (2.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 22 bytes 2418 (2.3 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
镜像：
[root@topcheer ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/mongo latest 965553e202a4 4 weeks ago 363 MB docker.io/rabbitmq management 4b23cfb64730 6 weeks ago 180 MB docker.io/sonatype/nexus3 latest 8eb898be2a53 7 weeks ago 611 MB docker.io/redis latest 01a52b3b5cd1 2 months ago 98.2 MB docker.io/nginx latest f949e7d76d63 2 months ago 126 MB wgr/centos latest fb7e3d506043 2 months ago 202 MB mytomcat9 latest 6c243064a028 2 months ago 749 MB myip 1.2 00a0a1f80e36 2 months ago 271 MB myip latest 420c99c3b707 2 months ago 271 MB mycentosfile 1.1 f022cd7b9017 2 months ago 395 MB topcher/tomcat 1.0.1 3d8737216a1e 2 months ago 508 MB docker.io/tomcat latest 8973f493aa0a 2 months ago 508 MB docker.io/mysql latest b8fd9553f1f0 2 months ago 445 MB docker.io/centos latest 67fa590cfc1c 3 months ago 202 MB docker.io/portainer/portainer latest 2b4ddf654e1c 4 months ago 77.7 MB docker.io/hello-world latest fce289e99eb9 11 months ago 1.84 kB docker.elastic.co/logstash/logstash 6.4.3 69c8a126e24a 13 months ago 674 MB docker.elastic.co/kibana/kibana 6.4.3 69271c2e80d6 13 months ago 724 MB docker.elastic.co/elasticsearch/elasticsearch 6.4.3 01e5bee1e059 13 months ago 795 MB [root@topcheer ~]#
启动ES
[root@topcheer config]# cat elasticsearch.yml network.host: 0.0.0.0 xpack: ml.enabled: false monitoring.enabled: false security.enabled: false watcher.enabled: false [root@topcheer config]# [root@topcheer ~]# docker run --rm -it -p 9200:9200 -p 9300:9300 --privileged=true -v /elk/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml docker.elastic.co/elasticsearch/elasticsearch:6.4.3 OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [2019-12-04T07:35:46,661][INFO ][o.e.n.Node ] [] initializing ... [2019-12-04T07:35:46,905][INFO ][o.e.e.NodeEnvironment ] [FWuG2IV] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [7.5gb], net total_space [26.9gb], types [rootfs] [2019-12-04T07:35:46,905][INFO ][o.e.e.NodeEnvironment ] [FWuG2IV] heap size [1007.3mb], compressed ordinary object pointers [true] [2019-12-04T07:35:46,908][INFO ][o.e.n.Node ] [FWuG2IV] node name derived from node ID [FWuG2IVRTuaar0I3USryyw]; set [node.name] to override [2019-12-04T07:35:46,909][INFO ][o.e.n.Node ] [FWuG2IV] version[6.4.3], pid[1], build[default/tar/fe40335/2018-10-30T23:17:19.084789Z], OS[Linux/3.10.0-1062.1.1.el7.x86_64/amd64], JVM["Oracle Corporation"/OpenJDK 64-Bit Server VM/10.0.2/10.0.2+13] [2019-12-04T07:35:46,909][INFO ][o.e.n.Node ] [FWuG2IV] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.bF3IHcYP, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [aggs-matrix-stats] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [analysis-common] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [ingest-common] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [lang-expression] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [lang-mustache] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [lang-painless] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [mapper-extras] [2019-12-04T07:35:52,613][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [parent-join] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [percolator] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [rank-eval] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [reindex] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [repository-url] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [transport-netty4] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [tribe] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-core] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-deprecation] [2019-12-04T07:35:52,614][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-graph] [2019-12-04T07:35:52,615][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-logstash] [2019-12-04T07:35:52,615][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-ml] [2019-12-04T07:35:52,616][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-monitoring] [2019-12-04T07:35:52,616][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-rollup] [2019-12-04T07:35:52,616][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-security] [2019-12-04T07:35:52,630][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-sql] [2019-12-04T07:35:52,630][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-upgrade] [2019-12-04T07:35:52,630][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded module [x-pack-watcher] [2019-12-04T07:35:52,631][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded plugin [ingest-geoip] [2019-12-04T07:35:52,631][INFO ][o.e.p.PluginsService ] [FWuG2IV] loaded plugin [ingest-user-agent] [2019-12-04T07:35:55,532][WARN ][o.e.d.s.ScriptModule ] Script: returning default values for missing document values is deprecated. Set system property ‘-Des.scripting.exception_for_missing_value=true‘ to make behaviour compatible with future major versions. [2019-12-04T07:35:59,406][INFO ][o.e.d.DiscoveryModule ] [FWuG2IV] using discovery type [zen] [2019-12-04T07:36:01,574][INFO ][o.e.n.Node ] [FWuG2IV] initialized [2019-12-04T07:36:01,574][INFO ][o.e.n.Node ] [FWuG2IV] starting ... [2019-12-04T07:36:02,058][INFO ][o.e.t.TransportService ] [FWuG2IV] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300} [2019-12-04T07:36:02,112][INFO ][o.e.b.BootstrapChecks ] [FWuG2IV] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-12-04T07:36:05,260][INFO ][o.e.c.s.MasterService ] [FWuG2IV] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {FWuG2IV}{FWuG2IVRTuaar0I3USryyw}{FmBH9fGtQo6mdG-uI_OwWA}{172.17.0.2}{172.17.0.2:9300}{xpack.installed=true} [2019-12-04T07:36:05,267][INFO ][o.e.c.s.ClusterApplierService] [FWuG2IV] new_master {FWuG2IV}{FWuG2IVRTuaar0I3USryyw}{FmBH9fGtQo6mdG-uI_OwWA}{172.17.0.2}{172.17.0.2:9300}{xpack.installed=true}, reason: apply cluster state (from master [master {FWuG2IV}{FWuG2IVRTuaar0I3USryyw}{FmBH9fGtQo6mdG-uI_OwWA}{172.17.0.2}{172.17.0.2:9300}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]]) [2019-12-04T07:36:05,461][INFO ][o.e.h.n.Netty4HttpServerTransport] [FWuG2IV] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200} [2019-12-04T07:36:05,461][INFO ][o.e.n.Node ] [FWuG2IV] started [2019-12-04T07:36:05,522][INFO ][o.e.g.GatewayService ] [FWuG2IV] recovered [0] indices into cluster_state
启动logstash
配置文件
[root@topcheer config]# cd logstash/ [root@topcheer logstash]# ll 总用量 0 drwxr-xr-x. 2 root root 47 9月 29 13:20 config drwxr-xr-x. 3 root root 44 9月 29 14:43 pipeline [root@topcheer logstash]# cd config/ [root@topcheer config]# ll 总用量 8 -rw-r--r--. 1 root root 118 9月 29 13:13 logstash.yml -rw-r--r--. 1 root root 85 9月 29 13:15 pipelines.yml [root@topcheer config]# cat logstash.yml config: reload: automatic: true interval: 3s xpack: management.enabled: false monitoring.enabled: false [root@topcheer config]# cat pipelines.yml - pipeline.id: test path.config: "/usr/share/logstash/pipeline/logstash-test.conf" [root@topcheer config]# [root@topcheer logstash]# ll 总用量 0 drwxr-xr-x. 2 root root 47 9月 29 13:20 config drwxr-xr-x. 3 root root 44 9月 29 14:43 pipeline [root@topcheer logstash]# [root@topcheer logstash]# cd pipeline/ [root@topcheer pipeline]# ll 总用量 4 drwxr-xr-x. 2 root root 22 9月 29 13:28 logs -rw-r--r--. 1 root root 193 9月 29 14:43 logstash-test.conf [root@topcheer pipeline]# cat logstash-test.conf input { file { path => ["/usr/share/logstash/pipeline/logs/test.log"] start_position => "beginning" } } output { elasticsearch { hosts => ["172.17.0.2:9200"] } }
[root@topcheer ~]# curl 172.17.0.2:9200 { "name" : "FWuG2IV", "cluster_name" : "elasticsearch", "cluster_uuid" : "6wMLPgpcRReweiZ4h7Yt5w", "version" : { "number" : "6.4.3", "build_flavor" : "default", "build_type" : "tar", "build_hash" : "fe40335", "build_date" : "2018-10-30T23:17:19.084789Z", "build_snapshot" : false, "lucene_version" : "7.4.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } [root@topcheer ~]# docker run --rm -it --privileged=true -v /elk/config/logstash/pipeline/:/usr/share/logstash/pipeline/ -v /elk/config/logstash/config/:/usr/share/logstash/config/ docker.elastic.co/logstash/logstash:6.4.3 Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO ] 2019-12-04 07:38:00.456 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"} [INFO ] 2019-12-04 07:38:00.501 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"} [INFO ] 2019-12-04 07:38:00.885 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"a90dce9b-7a35-47d0-a15b-c26104c52fd0", :path=>"/usr/share/logstash/data/uuid"} [INFO ] 2019-12-04 07:38:01.100 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.4.3"} [INFO ] 2019-12-04 07:38:01.972 [Converge PipelineAction::Create<test>] pipeline - Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50} [INFO ] 2019-12-04 07:38:02.389 [[test]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.17.0.2:9200/]}} [INFO ] 2019-12-04 07:38:02.395 [[test]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://172.17.0.2:9200/, :path=>"/"} [WARN ] 2019-12-04 07:38:02.563 [[test]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://172.17.0.2:9200/"} [INFO ] 2019-12-04 07:38:03.087 [[test]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6} [WARN ] 2019-12-04 07:38:03.087 [[test]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won‘t be used to determine the document _type {:es_version=>6} [INFO ] 2019-12-04 07:38:03.095 [Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:42] elasticsearch - Using mapping template from {:path=>nil} [INFO ] 2019-12-04 07:38:03.098 [[test]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.17.0.2:9200"]} [INFO ] 2019-12-04 07:38:03.140 [Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:42] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}} [INFO ] 2019-12-04 07:38:03.231 [Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:42] elasticsearch - Installing elasticsearch template to _template/logstash [INFO ] 2019-12-04 07:38:03.512 [[test]>worker1] file - No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_dfd0c462141bf3c141674072493f4e56", :path=>["/usr/share/logstash/pipeline/logs/test.log"]} [INFO ] 2019-12-04 07:38:03.603 [Converge PipelineAction::Create<test>] pipeline - Pipeline started successfully {:pipeline_id=>"test", :thread=>"#<Thread:0x1e9fb835@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:157 run>"} [INFO ] 2019-12-04 07:38:03.660 [[test]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections [INFO ] 2019-12-04 07:38:03.755 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :running_pipelines=>[:test], :non_running_pipelines=>[]} [INFO ] 2019-12-04 07:38:04.144 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
启动kibana
[root@topcheer config]# cat kibana.yml server.host: "0.0.0.0" elasticsearch.url: http://172.17.0.2:9200 xpack: apm.ui.enabled: false graph.enabled: false ml.enabled: false monitoring.enabled: false reporting.enabled: false security.enabled: false grokdebugger.enabled: false searchprofiler.enabled: false
[root@topcheer ~]# docker run --rm -it -p 5601:5601 --privileged=true -v /elk/config/kibana.yml:/usr/share/kibana/config/kibana.yml docker.elastic.co/kibana/kibana:6.4.3 log [07:39:57.733] [info][optimize] Optimizing and caching bundles for stateSessionStorageRedirect, status_page, timelion, dashboardViewer, apm and kibana. This may take a few minutes
kibana启动的时间要很久
之前初始化1个多小时都没有好，之前尝试过只要40分钟就可以了,此处就不等了，可以自己测一下
以上是关于docker安装ELK的主要内容，如果未能解决你的问题，请参考以下文章