elk安装部署
Posted yangxiaochu
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了elk安装部署相关的知识，希望对你有一定的参考价值。
环境介绍：
服务端：192.168.0.248 ：（es ,logstash,kibana,redis）
客户端：192.168.0.151 192.168.0.219
一：服务器端配置 192.168.0.248：
1）安装es:
https://www.cnblogs.com/yangxiaochu/p/9894536.html
2)安装redis
1 cd /opt 2 wget http://download.redis.io/releases/redis-4.0.1.tar.gz 3 tar xzf redis-4.0.1.tar.gz 4 cd redis-4.0.1 5 make
1 [[email protected] redis-4.0.1]# ls 2 00-RELEASENOTES BUGS CONTRIBUTING COPYING deps INSTALL Makefile MANIFESTO README.md redis.conf runtest runtest-cluster runtest-sentinel sentinel.conf src tests utils 3 [[email protected] redis-4.0.1]# cd src/ 4 [[email protected] src]# make install PREFIX=/usr/local/redis 5 CC Makefile.dep 6 7 Hint: It‘s a good idea to run ‘make test‘ ;) 8 9 INSTALL install 10 INSTALL install 11 INSTALL install 12 INSTALL install 13 INSTALL install 14 [[email protected] redis-4.0.1]# cp redis.conf /usr/local/redis/ 15 [[email protected] redis-4.0.1]# cd /usr/local/redis/ 16 [[email protected] redis]# ls 17 bin redis.conf 18 [[email protected] redis]# /usr/local/redis/bin/redis-se 19 redis-sentinel redis-server 20 [[email protected] redis]# /usr/local/redis/bin/redis-server /usr/local/redis/redis.conf 21 6929:C 02 Nov 10:12:41.884 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 22 6929:C 02 Nov 10:12:41.885 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=6929, just started 23 6929:C 02 Nov 10:12:41.885 # Configuration loaded 24 _._ 25 _.-``__ ‘‘-._ 26 _.-`` `. `_. ‘‘-._ Redis 4.0.1 (00000000/0) 64 bit 27 .-`` .-```. ```/ _.,_ ‘‘-._ 28 ( ‘ , .-` | `, ) Running in standalone mode 29 |`-._`-...-` __...-.``-._|‘` _.-‘| Port: 6379 30 | `-._ `._ / _.-‘ | PID: 6929 31 `-._ `-._ `-./ _.-‘ _.-‘ 32 |`-._`-._ `-.__.-‘ _.-‘_.-‘| 33 | `-._`-._ _.-‘_.-‘ | http://redis.io 34 `-._ `-._`-.__.-‘_.-‘ _.-‘ 35 |`-._`-._ `-.__.-‘ _.-‘_.-‘| 36 | `-._`-._ _.-‘_.-‘ | 37 `-._ `-._`-.__.-‘_.-‘ _.-‘ 38 `-._ `-.__.-‘ _.-‘ 39 `-._ _.-‘ 40 `-.__.-‘ 41 42 6929:M 02 Nov 10:12:41.887 # Server initialized 43 6929:M 02 Nov 10:12:41.887 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run 44 the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled‘ as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.6929:M 02 Nov 10:12:41.887 * Ready to accept connections 45 46 47 ^C6929:signal-handler (1541124774) Received SIGINT scheduling shutdown... 48 6929:M 02 Nov 10:12:54.552 # User requested shutdown... 49 6929:M 02 Nov 10:12:54.553 * Saving the final RDB snapshot before exiting. 50 6929:M 02 Nov 10:12:54.554 * DB saved on disk 51 6929:M 02 Nov 10:12:54.554 * Removing the pid file. 52 6929:M 02 Nov 10:12:54.554 # Redis is now ready to exit, bye bye...
编辑配置文件
1 [[email protected] redis]# vim redis.conf 2 # 将daemonize的值改为yes 默认情况，Redis不是在后台运行，我们需要把redis放在后台运行 3 # By default Redis does not run as a daemon. Use ‘yes‘ if you need it. 4 # Note that Redis will write a pid file in /var/run/redis.pid when daemonized. 5 daemonize yes 6 bind 0.0.0.0
重新启动redis
1 [[email protected] redis]# /usr/local/redis/bin/redis-server /usr/local/redis/redis.conf 2 6935:C 02 Nov 10:14:19.229 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 3 6935:C 02 Nov 10:14:19.229 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=6935, just started 4 6935:C 02 Nov 10:14:19.229 # Configuration loaded
测试redis
1 [[email protected] redis]# /usr/local/redis/bin/redis-cli 2 127.0.0.1:6379> set aa aavlue 3 OK 4 127.0.0.1:6379> get aa 5 "aavlue" 6 127.0.0.1:6379> exit
3)安装logstash
1 cd /opt 2 wget https://artifacts.elastic.co/downloads/logstash/logstash-5.5.2.tar.gz 3 tar -xzf logstash-5.5.2.tar.gz
测试一下：
测试logstash 是否正常运行
1 /opt/logstash-5.5.2/bin/logstash -e ‘input{stdin{}}output{stdout{codec=>rubydebug}}‘
敲入Hello World,回车
输出结果：
Hello World { "@timestamp" => 2017-08-30T17:24:25.553Z, "@version" => "1", "host" => "node4", "message" => "Hello World" }
开启logstash indexer
1 cd /opt/logstash-5.5.2 2 mkdir etc 3 vim /opt/logstash-5.5.2/etc/logstash_indexer.conf
1 input { 2 redis { 3 host => "192.168.0.248" 4 data_type => "list" 5 port => "6379" 6 key => "logstash:redis" 7 type => "redis-input" 8 } 9 } 10 11 output { 12 elasticsearch { 13 hosts => ["192.168.0.248:9200"] 14 index => "logstash-%{type}-%{+YYYY.MM.dd}" 15 document_type => "%{type}" 16 flush_size => 20000 17 idle_flush_time => 10 18 sniffing => true 19 template_overwrite => true 20 } 21 }
input {}解释
读取Redis key logstash:redis 的数据
output {}解释
批量发送Elasticsearch，本插件的 flush_size 和 idle_flush_time 两个参数共同控制 Logstash 向 Elasticsearch 发送批量数据的行为。以上面示例来说：Logstash 会努力攒到 20000 条数据一次性发送出去，但是如果 10 秒钟内也没攒够 20000 条，Logstash 还是会以当前攒到的数据量发一次。 默认情况下，flush_size 是 500 条，idle_flush_time 是 1 秒。这也是很多人改大了 flush_size 也没能提高写入 ES 性能的原因——Logstash 还是 1 秒钟发送一次。
启动logstash
cd /opt/logstash-5.5.2/logs/ mkdir log1 log2
启动 logstash indexer logstash 读redis日志发送到es
1 nohup /opt/logstash-5.5.2/bin/logstash -f /opt/logstash-5.5.2/etc/logstash_indexer.conf --path.data=/opt/logstash-5.5.2/logs/log2 > /dev/null 2>&1 &
4)安装kibana
1 [email protected] opt]# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-linux-x86_64.tar.gz 2 --2018-11-02 10:35:24-- https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-linux-x86_64.tar.gz 3 Resolving artifacts.elastic.co (artifacts.elastic.co)... 107.21.202.15, 107.21.237.188, 107.21.237.95, ... 4 Connecting to artifacts.elastic.co (artifacts.elastic.co)|107.21.202.15|:443... connected. 5 HTTP request sent, awaiting response... 200 OK 6 Length: 51073441 (49M) [binary/octet-stream] 7 Saving to: ‘kibana-5.5.2-linux-x86_64.tar.gz’ 8 9 100%[===================================================================================================================================================>] 51,073,441 274KB/s in 3m 41s 10 11 2018-11-02 10:39:06 (226 KB/s) - ‘kibana-5.5.2-linux-x86_64.tar.gz’ saved [51073441/51073441] 12 13 [[email protected] opt]# ls 14 elasticsearch-5.5.2 jdk1.8.0_102 kibana-5.5.2-linux-x86_64.tar.gz logstash-5.5.2.tar.gz redis-4.0.1.tar.gz 15 elasticsearch-5.5.2.tar.gz jdk-8u102-linux-x64.tar.gz logstash-5.5.2 redis-4.0.1 16 [[email protected] opt]# tar xf kibana-5.5.2-linux-x86_64.tar.gz 17 [[email protected] opt]# ls 18 elasticsearch-5.5.2 jdk1.8.0_102 kibana-5.5.2-linux-x86_64 logstash-5.5.2 redis-4.0.1 19 elasticsearch-5.5.2.tar.gz jdk-8u102-linux-x64.tar.gz kibana-5.5.2-linux-x86_64.tar.gz logstash-5.5.2.tar.gz redis-4.0.1.tar.gz 20 [[email protected] opt]# cd kibana-5.5.2-linux-x86_64 21 [[email protected] kibana-5.5.2-linux-x86_64]# ls 22 bin config data LICENSE.txt node node_modules NOTICE.txt optimize package.json plugins README.txt src ui_framework webpackShims 23 [[email protected] kibana-5.5.2-linux-x86_64]# vim config/kibana.yml 24 [[email protected] kibana-5.5.2-linux-x86_64]# cd bin/ 25 [[email protected] bin]# ls 26 kibana kibana-plugin 27 [[email protected] bin]# ./kibana 28 29 log [02:41:35.783] [info][status][plugin:[email protected]5.5.2] Status changed from uninitialized to green - Ready 30 log [02:41:35.873] [info][status][plugin:[email protected]5.5.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch 31 log [02:41:35.908] [info][status][plugin:[email protected]5.5.2] Status changed from uninitialized to green - Ready 32 log [02:41:35.945] [info][status][plugin:[email protected]5.5.2] Status changed from uninitialized to green - Ready 33 log [02:41:36.137] [info][status][plugin:[email protected]5.5.2] Status changed from uninitialized to green - Ready 34 log [02:41:36.142] [info][listening] Server running at http://0.0.0.0:5601 35 log [02:41:36.143] [info][status][ui settings] Status changed from uninitialized to yellow - Elasticsearch plugin is yellow 36 log [02:41:40.958] [info][status][plugin:[email protected]5.5.2] Status changed from yellow to yellow - No existing Kibana index found 37 log [02:41:41.453] [info][status][plugin:[email protected]5.5.2] Status changed from yellow to green - Kibana index ready 38 log [02:41:41.456] [info][status][ui settings] Status changed from yellow to green - Ready
编辑配置文件修改
server.port: 5601 server.host: "0.0.0.0" elasticsearch.url: "http://192.168.0.248:9200"
客户端配置
1）安装logstash
安装过程同上
编辑logstash收集日志发送给redis的配置文件
input {
file {
type => "nginx_access_log"
path => "/usr/local/nginx/logs/host.access.log"
}
}
output {
redis {
host => "192.168.0.248"
data_type => "list"
port => "6379"
key => "logstash:redis"
}
}
启动logstash agent
[[email protected] log1]# nohup /opt/logstash-5.5.2/bin/logstash -f /opt/logstash-5.5.2/etc/logstash_agent_nginx.conf --path.data=/opt/logstash-5.5.2/logs/log1 > /dev/null 2>&1
kibana页面配置
以上是关于elk安装部署的主要内容，如果未能解决你的问题，请参考以下文章