ansible playbook部署ELK集群系统
Posted wang_zai
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ansible playbook部署ELK集群系统相关的知识，希望对你有一定的参考价值。
一、介绍
总共4台机器，分别为
192.168.1.99
192.168.1.100
192.168.1.210
192.168.1.211
服务所在机器为：
redis：192.168.1.211
logstash: 192.168.210 192.168.1.211
elasticsearch: 分别布置在四台（211不为主、不存储数据）
kibana：192.168.1.211
nginx：192.168.1.211
二、elk集群架构图
三、为每一个软件或集群创建一个单独的角色。
cd /etc/ansible
mkdir -pv elk/roles/{jdk,redis,logstash,es,es_head,kibana,nginx}/{tasks,files,templates,meta,handlers,vars}
四、配置hosts
在/etc/ansible/hosts添加如下内容：
[jdk_servers] 192.168.1.99 192.168.1.100 192.168.1.210 192.168.1.211 [redis_servers] 192.168.1.211 [es_servers] 192.168.1.99 node_name=node-99 node_master=true node_data=true network_host=192.168.1.99 192.168.1.100 node_name=node-100 node_master=true node_data=true network_host=192.168.1.100 192.168.1.210 node_name=node-210 node_master=true node_data=true network_host=192.168.1.210 192.168.1.211 node_name=node-211 node_master=true node_data=true network_host=192.168.1.211 [es_plugin_head] 192.168.1.211 [logstash_servers] 192.168.1.210 192.168.1.211 [kibana_servers] 192.168.1.211 [nginx_servers] 192.168.1.211
PS：es_servers组里面，后面定义了变量，因为es配置文件中，会根据服务器的不同，对应的配置也不同，在这里定义，可以直接在tasks里面去引用
五、JDK roles
1.把jdk-8u91-linux-x64.tar.gz和Java环境变量脚本传到files目录下
2.编写vars
[root@master jdk]# cat vars/main.yml jdk_package_name: jdk-8u91-linux-x64.tar.gz env_dir: /etc/profile.d
3.编写tasks
[root@master jdk]# cat tasks/main.yml - name: copy jdk software copy: src={{ jdk_package_name }} dest=/tmp/ owner=root group=root - name: uncompression jdk software shell: tar -zxf /tmp/{{ jdk_package_name }} -C /usr/local/ - name: copy jdk env copy: src=java.sh dest={{ env_dir }}/ owner=root group=root mode=0644 - name: enforce env shell: source {{ env_dir }}/java.sh - name: delete jdk compressin files shell: rm -f /tmp/{{ jdk_package_name }}
4.使用角色
在roles同级目录，创建一个jdk.yml文件，里面定义好你的playbook。
[root@master elk]# cat jdk.yml - hosts: 127.0.0.1 remote_user: root roles: - jdk
5. 运行playbook安装jdk
[root@master elk]# ansible-playbook jdk.yml
六、redis roles
1.把redis.tar.gz传到files目录下，redis配置文件和启动脚本传到templates目录下
2.编写vars
[root@master redis]# cat vars/main.yml redis_package_file: redis.tar.gz redis_dir: /usr/local/redis exec_dir: /etc/init.d
3.编写tasks
[root@master redis]# cat tasks/main.yml - name: copy redis software copy: src={{ redis_package_file }} dest=/tmp/ owner=root group=root - name: uncompression redis software shell: tar zxf /tmp/{{ redis_package_file }} -C /usr/local - name: copy redis start script template: src=redis dest={{ exec_dir }}/ owner=root group=root mode=0755 - name: copy redis config template: src=redis.conf dest={{ redis_dir }}/etc/ owner=redis group=redis - name: create redis user user: name=redis shell=/sbin/nologin - name: change owner and group file: path={{ redis_dir }} owner=redis group=redis #recurse=yes - name: mkdir directory for redis data file: dest=/data/redis mode=0755 state=directory owner=redis group=redis - name: start redis service shell: /etc/init.d/redis start - name: add boot start redis start shell: chkconfig --level 345 redis on - name: delete redis compression files shell: rm -f /tmp/{{ redis_package_file }}
4.使用角色
在roles同级目录，创建一个redis.yml文件，里面定义好你的playbook
[root@master elk]# cat redis.yml - hosts: redis_servers remote_user: root roles: - redis
5. 运行playbook安装redis
[root@master elk]# ansible-playbook redis.yml
七、logstash roles
1.把logstash.tar.gz传到files目录下，logstash自定义配置文件和启动脚本传到templates目录下
2.编写vars
[root@master logstash]# cat vars/main.yml logstash_package_file: logstash.tar.gz logstash_dir: /usr/local/logstash
3.编写tasks
[root@master logstash]# cat tasks/main.yml - name: copy logstash software copy: src={{ logstash_package_file }} dest=/tmp/ owner=root group=root - name: uncompression logstash software shell: tar zxf /tmp/{{ logstash_package_file }} -C /usr/local - name: mkdir config files dir file: dest={{ logstash_dir }}/conf.d mode=0755 state=directory owner=es group=es - name: copy logstash start script template: src=start.sh dest={{ logstash_dir }}/ owner=es group=es mode=0755 - name: copy logstash config template: src=visit.conf dest={{ logstash_dir }}/conf.d/ owner=es group=es - name: create es user user: name=es - name: change owner and group file: path={{ logstash_dir }} owner=es group=es - name: start logstash service shell: su - es -c "{{ logstash_dir }}/start.sh" - name: delete logstash compression files shell: rm -f /tmp/{{ logstash_package_file }}
4.使用角色
在roles同级目录，创建一个logstash.yml文件，里面定义好你的playbook
[root@master elk]# cat logstash.yml - hosts: logstash_servers remote_user: root roles: - logstash
5. 运行playbook安装logstash
[root@master elk]# ansible-playbook logstash.yml
八、es roles
1.把elasticsearch.tar.gz传到files目录下，es配置文件和启动脚本传到templates目录下
2.编写vars
[root@master es]# cat vars/main.yml es_package_file: elasticsearch.tar.gz es_dir: /usr/local/elasticsearch config_name: elasticsearch.yml
3.编写tasks
[root@master es]# cat tasks/main.yml - name: copy es software copy: src={{ es_package_file }} dest=/tmp/ owner=root group=root - name: uncompression es software shell: tar zxf /tmp/{{ es_package_file }} -C /usr/local - name: copy es start script template: src=start.sh dest={{ es_dir }}/ owner=es group=es mode=0755 - name: copy es config template: src=elasticsearch.yml dest={{ es_dir }}/config/ owner=es group=es - name: update es config shell: sed -i "{{ item }}" {{ es_dir }}/config/{{ config_name }} with_items: - \'s/#node.name/node.name: {{ node_name }}/\' - \'s/#node.master/node.master: {{ node_master }}/\' - \'s/#node.data/node.data: {{ node_data }}/\' - \'s/#network.host/network.host: {{ network_host }}/\' - name: create es user user: name=es - name: change owner and group file: path={{ es_dir }} owner=es group=es #recurse=yes - name: mkdir directory for es data file: dest=/data/elasticsearch mode=0755 state=directory owner=es group=es notify: mkdir Subdirectory - name: start es service shell: su es -c \'cd {{ es_dir }} && ./start.sh\' - name: delete es compression files shell: rm -f /tmp/{{ es_package_file }}
4.编写handlers
[root@master es]# cat handlers/main.yml - name: mkdir Subdirectory file: dest=/data/elasticsearch/{{ item }} mode=0755 state=directory owner=es group=es with_items: - logs - data
5.使用角色
在roles同级目录，创建一个es.yml文件，里面定义好你的playbook
[root@master elk]# cat es.yml - hosts: es_servers remote_user: root roles: - es
5. 运行playbook安装es
[root@master elk]# ansible-playbook es.yml
九、es-head roles
1.把node-v4.4.7-linux-x64.tar.gz和elasticsearch-head.tar.gz传到files目录下，把_site/app.js传到templates目录下
2.编写vars
[root@master es_head]# cat vars/main.yml node_package_file: node-v4.4.7-linux-x64.tar.gz node_dir: /usr/local/node-v4.4.7-linux-x64 head_package_file: elasticsearch-head.tar.gz head_dir: /usr/local/elasticsearch/elasticsearch-head
3.编写tasks
[root@master es_head]# cat tasks/main.yml - name: copy node software copy: src={{ node_package_file }} dest=/tmp/ owner=root group=root - name: uncompression node software shell: tar zxf /tmp/{{ node_package_file }} -C /usr/local/ - name: copy node to env shell: rsync -a {{ node_dir }}/bin/* /usr/bin/ - name: copy head plugin software copy: src={{ head_package_file }} dest=/tmp/ owner=root group=root - name: uncopmression head plugin software shell: tar zxf /tmp/{{ head_package_file }} -C /usr/local/elasticsearch/ - name: copy head config template: src=app.js dest={{ head_dir }}/_site/ owner=es group=es - name: change owner and group file: path={{ head_dir }} owner=es group=es #recurse=yes - name: start es service shell: su es -c \'cd {{ head_dir }} && npm run start > {{ head_dir }}/run.log &\' - name: delete node compression files shell: rm -f /tmp/{{ node_package_file }} - name: delete head compression files shell: rm -f /tmp/{{ head_package_file }}
4.使用角色
在roles同级目录，创建一个es_head.yml文件，里面定义好你的playbook
[root@master elk]# cat es_head.yml - hosts: es_plugin_head remote_user: root roles: - es_head
5. 运行playbook安装es-head
ansible-playbook es_head.yml
十、kibana roles
1.把kibana.tar.gz传到files目录下，kibana配置文件和启动脚本传到templates目录下
2.编写vars
[root@master kibana]# cat vars/main.yml kibana_package_file: kibana.tar.gz kibana_dir: /usr/local/kibana
3.编写tasks
[root@master kibana]# cat tasks/main.yml - name: copy kibana software copy: src={{ kibana_package_file }} dest=/tmp/ owner=root group=root - name: uncompression kibana software shell: tar zxf /tmp/{{ kibana_package_file }} -C /usr/local - name: copy kibana start script template: src=start.sh dest={{ kibana_dir }}/ owner=es group=es mode=0755 - name: copy kibana config template: src=kibana.yml dest={{ kibana_dir }}/config/ owner=es group=es - name: change owner and group file: path={{ kibana_dir }} owner=es group=es - name: start kibana service shell: su - es -c "{{ kibana_dir }}/start.sh" - name: delete kibana compression files shell: rm -f /tmp/{{ kibana_package_file }}
4.使用角色
在roles同级目录，创建一个kibana.yml文件，里面定义好你的playbook
[root@master elk]# cat kibana.yml - hosts: kibana_servers remote_user: root roles: - kibana
5. 运行playbook安装kibana
ansible-playbook kibana.yml
十一、nginx roles（211本身就已经安装nginx,如果目标机器没安装，请参考我之前的文章）
1.把htpasswd生成密码的对应文件传到files目录下，nginx配置文件传到templates目录下
2.编写vars
[root@master nginx]# cat vars/main.yml nginx_basedir: /usr/local/nginx
3.编写tasks
[root@master nginx]# cat tasks/main.yml - name: copy conf file template: src={{ item.src }} dest={{ nginx_basedir }}/{{ item.dest }} backup=yes owner=root group=root mode=0644 with_items: - { src: es.conf, dest: conf/vhosts/ } - { src: kibana.conf, dest: conf/vhosts/ } - name: copy passwd file copy: src={{ item }} dest={{ nginx_basedir }}/conf/htpasswd/ owner=root group=root mode=0644 with_items: - es - kibana notify: restart nginx
4.编写handlers
[root@master nginx]# cat handlers/main.yml - name: restart nginx shell: /usr/local/nginx/sbin/nginx -s reload
5.使用角色
在roles同级目录，创建一个nginx.yml文件，里面定义好你的playbook
[root@master elk]# cat nginx.yml --- - hosts: nginx_servers user: root roles: - nginx
5. 运行playbook加载nginx
ansible-playbook nginx.yml
以上是关于ansible playbook部署ELK集群系统的主要内容，如果未能解决你的问题，请参考以下文章