Elastic:如何使用 Ansible自动化部署 Elastic Stack -Security
Posted Elastic 中国社区官方博客
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Elastic:如何使用 Ansible自动化部署 Elastic Stack -Security相关的知识,希望对你有一定的参考价值。
在之前我的系列文章:
我们已经学习了如何使用 Ansible 来部署我们的 Elasticsearch 以及 Kibana。在进行下面的练习之前,我希望你先做完上面的三个练习。在实际的生产环境中,我们会使用 Elasticsearch 的安全。为内置的用户配置密码。如果你对如何为 Elasticsearch 及 Kibana 配置安全还不是很熟悉的话,请参阅我之前的文章:
在上面,我展示了两种配置安全的方法。第一种方法比较适合只有很少服务器的情况,第二种方式更适合脚本或者像 ansible 这样的工具进行部署。在今天的教程中,我将使用第二种方法来进行部署。
你可以在地址找到源码:https://github.com/liu-xiao-guo/elk-ansible
为 Elastic Stack 配置安全
定义内置用户
在下面,我们将一步一步地指导如何为 Elasticsearch 及 Kibana 配置上安全。如果你对 Elastic Stack 的内置用户还不是很熟的话,请阅读我们的官方文档 “内置用户”。为了能够创建这些内置用户,我们在之前的文件架构中,我们创建一个新的文件目录 vars:
$ pwd
/Users/liuxg/ansible/elasticsearch
$ mkdir vars
$ ls
inventory playbooks roles vars
在这个 vars 目录中,我们创建一个新的文件 credentials.yml:
vars/credentials.yml
# Bootstrap credetials
es_api_basic_auth_username: elastic
es_api_basic_auth_password: s3cr3t
# A built-in superuser.
elastic_username: elastic
elastic_password: goodwitch
# The user Kibana uses to connect and communicate with Elasticsearch.
kibana_username: kibana
kibana_password: badsanta
# The user Logstash uses when storing monitoring information in Elasticsearch.
logstash_system_username: logstash_system
logstash_system_password: dragonprince
# The user the Beats use when storing monitoring information in Elasticsearch.
beats_system_username: beats_system
beats_system_password: avatar
# The user the APM server uses when storing monitoring information in Elasticsearch.
apm_system_username: apm_system
apm_system_password: mashaandthebear
# The user Metricbeat uses when collecting and storing monitoring information in Elasticsearch. It has the remote_monitoring_agent and remote_monitoring_collector built-in roles.
remote_monitoring_user_username: remote_monitoring_user
remote_monitoring_user_password: gossipgirl
在上面我们定义了 Elastic Stack 的一些内置用户的用户名及密码。这些变量将会被 Elastic Stack 所有软件所使用,所以它独立于任何一个 roles。
同时,我们定义一些常用的变量。在这个 vars 目录中,我们来创建一个叫做 main.yml 的文件:
vars/main.yml
elastic_host: 192.168.0.4
elastic_port: 9200
kibana_host: 192.168.0.4
kibana_port: 5601
elastic_protocol: http
filebeat_http_port: 5067
auditbeat_http_port: 5068
heartbeat_http_port: 5069
packetbeat_http_port: 5070
web_server_1: 192.168.0.4
由于引入了上面的两个文件,我们需要重新修改我们之前定义的 deploy-demo.yml 文件:
playbooks/deploy-demo.yml
---
# This playbook will deploy webserver
- hosts: all
become: yes
roles:
- ../roles/add-elastic-repo
# This playbook will deploy ELK stack
- hosts: elk
become: yes
vars_files:
- ../vars/credentials.yml
- ../vars/main.yml
roles:
- ../roles/elasticsearch
- ../roles/kibana
启动安全
我们找到 templates/elasticsearch.yml 文件,并添加如下的项:
xpack.security.enabled: true
我们针对 elasticsearch.yml 文件做如下的修改:
templates/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: cluster_name
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node_name
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: path_data
#
# Path to log files:
#
path.logs: path_logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: network_host
#
# Set a custom port for HTTP:
#
http.port: http_port
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
discovery.type: discovery_type
# -------------------------------- X-pack security Configuration ----------------
xpack.security.enabled: true
为了方便我们的设计,我们修改 elasticsearch 角色下的 defaults/main.yml 文件,并定义如下的变量:
elasticsearch/defaults/main.yml
---
# defaults file for elasticsearch
cluster_name: demo-elk
node_name: elk-1
path_data: /var/lib/elasticsearch
path_logs: /var/log/elasticsearch
network_host: 0.0.0.0
http_port: 9200
discovery_type: single-node
# X-pack configuration
xpack_security_enabled: true
es_home: /usr/share/elasticsearch
es_conf_dir: /etc/elasticsearch
es_group: elasticsearch
es_owner: root
es_mode: 0660
es_api_protocol: "http"
es_api_host: "localhost"
es_api_port: 9200
es_api_uri: " es_api_protocol :// es_api_host : es_api_port "
es_security_api: _security
es_validate_certs: no
es_force_basic_auth: yes
在上面,我添加了 x-pack configuration 这个部分。如果你对这个部分的变量定义还不是很熟的话,请参考我之前的文章 “使用 elasticsearch-keystore 配置安全并创建内置用户账号”。在这里 es_home 定义了 Elasticsearch 被安装的目录,而 es_conf_dir 定义了配置文件的目录。es_api_url 定了如何使用 REST API 接口的方式来配置内置用户的接口。
我们接下来在 elasticsearch role 下的 tasks 里创建一个文件夹 security,并在这个文件夹里创建两个文件 keystore.yml 及 setup-built-in-user.yml:
$ pwd
/Users/liuxg/ansible/elasticsearch/roles/elasticsearch/tasks
$ tree -L 2
.
├── main.yml
└── security
├── keystore.yml
└── setup-built-in-user.yml
我们先来介绍 keystore.yml 这个文件:
keystore.yml
---
# ----------- Create KeyStore ----------
- name: create elasticsearch keystore
become: yes
command: >
es_home /bin/elasticsearch-keystore create
args:
creates: " es_conf_dir /elasticsearch.keystore"
environment:
ES_PATH_CONF: " es_conf_dir "
- name: Set elasticsearch keystore permissions
become: yes
file: state=file path= es_conf_dir /elasticsearch.keystore owner= es_owner group= es_group mode= es_mode
- name: Check if elasticsearch keystore is setup
become: yes
command: >
es_home /bin/elasticsearch-keystore list
register: list_keystore
changed_when: False
environment:
ES_PATH_CONF: " es_conf_dir "
check_mode: no
- name: Create bootstrap password for elastic user
become: yes
shell: echo " es_api_basic_auth_password " | es_home /bin/elasticsearch-keystore add -x 'bootstrap.password'
when:
- es_api_basic_auth_username is defined and list_keystore is defined and es_api_basic_auth_username == 'elastic' and 'bootstrap.password' not in list_keystore.stdout_lines
environment:
ES_PATH_CONF: " es_conf_dir "
no_log: true
# ------------including builtin user setup --------------
- name: Including builtin user setup
include: setup-built-in-user.yml
在上面的几个部分中,我在文章 “使用 elasticsearch-keystore 配置安全并创建内置用户账号” 分别有介绍。如果你不是很了解的话,建议你先去阅读那篇文章。
在上面的 keystore.yml 文件中,我们也有一处 include: setup-built-in-user.yml。setup-built-in-user.yml 的定义如下:
setup-built-in-user.yml
---
- name: Restarting Elasticsearch
service:
name: elasticsearch
state: restarted
#------------------------------- Setup Built-in User passwords-----------
- name: Update elastic user password
uri:
url: " es_api_uri / es_security_api /user/es_api_basic_auth_username/_password"
method: POST
body_format: json
body: " \\"password\\":\\" elastic_password \\" "
status_code: 200
user: "es_api_basic_auth_username"
password: "es_api_basic_auth_password"
force_basic_auth: " es_force_basic_auth "
validate_certs: " es_validate_certs "
- name: Update Kibana user password
uri:
url: " es_api_uri / es_security_api /user/ kibana_username /_password"
method: POST
body_format: json
body: " \\"password\\":\\" kibana_password \\" "
status_code: 200
user: "es_api_basic_auth_username"
password: " elastic_password "
force_basic_auth: " es_force_basic_auth "
validate_certs: " es_validate_certs "
- name: Update logstash_system user password
uri:
url: " es_api_uri / es_security_api /user/ logstash_system_username /_password"
method: POST
body_format: json
body: " \\"password\\":\\" logstash_system_password \\" "
status_code: 200
user: "es_api_basic_auth_username"
password: " elastic_password "
force_basic_auth: " es_force_basic_auth "
validate_certs: " es_validate_certs "
- name: Update beats_system user password
uri:
url: " es_api_uri / es_security_api /user/ beats_system_username /_password"
method: POST
body_format: json
body: " \\"password\\":\\" beats_system_password \\" "
status_code: 200
user: "es_api_basic_auth_username"
password: " elastic_password "
force_basic_auth: " es_force_basic_auth "
validate_certs: " es_validate_certs "
- name: Update APM system user password
uri:
url: " es_api_uri / es_security_api /user/ apm_system_username /_password"
method: POST
body_format: json
body: " \\"password\\":\\" apm_system_password \\" "
status_code: 200
user: "es_api_basic_auth_username"
password: " elastic_password "
force_basic_auth: " es_force_basic_auth "
validate_certs: " es_validate_certs "
- name: Update remote_monitoring_user password
uri:
url: " es_api_uri / es_security_api /user/ remote_monitoring_user_username /_password"
method: POST
body_format: json
body: " \\"password\\":\\" remote_monitoring_user_password \\" "
status_code: 200
user: "es_api_basic_auth_username"
password: " elastic_password "
force_basic_auth: " es_force_basic_auth "
validate_certs: " es_validate_certs "
这个部分也是非常直接。它直接使用了 REST API 接口。它们的使用在我之前的文章 “使用 elasticsearch-keystore 配置安全并创建内置用户账号” 已经做了展示。
我们需要更进一步修改 tasks 目录下的 main.yml 文件使得刚才被创建的 keystore.yml 及 setup-built-in-user.yml 被引用:
elasticsearch/tasks/main.yml
---
# tasks file for elasticsearch
# Installing Elasticsearch
- name: Installing Elasticsearch
apt:
name: elasticsearch
# Replce default elasticsearch.yml
- name: Replace default elasticsearch.yml
template:
src: elasticsearch.yml
dest: /etc/elasticsearch/elasticsearch.yml
# Start Elasticsearch service
- name:
service:
name: elasticsearch
state: started
enabled: yes
# Xpack security configuration
- name: Including xpack security setup configuration
include: security/keystore.yml
好了,到目前为止,我们已经完成了对 Elasticsearch 配置的要求。我们接下来需要对 Kibana 也做相应的修改。我们需要对 kibana.yml 文件添加如下的部分:
elasticsearch.username: " kibana_username "
elasticsearch.password: " kibana_password "
kibana/templates/kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: server_port
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: " server_host "
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name. This is used for display purposes.
server.name: " server_name "
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: [" elasticsearch_host "]
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: " kibana_username "
elasticsearch.password: " kibana_password "
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders:
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false
# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid
# Enables you to specify a file where Kibana stores log output.
#logging.dest: stdout
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
经过这样的配置后,我们可以使用 kibana 的用户名及密码进行登录了。
好了,一切准备就绪。我们再次使用如下的命令来进行部署:
ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-demo.yml
上面命令运行的结果是:
$ pwd
/Users/liuxg/ansible/elasticsearch
$ ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-demo.yml
BECOME password:
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.0.4]
TASK [../roles/add-elastic-repo : add elasticsearch public signing key] ********
ok: [192.168.0.4]
TASK [../roles/add-elastic-repo : Install apt-transport-https] *****************
ok: [192.168.0.4]
TASK [../roles/add-elastic-repo : Add elasticsearch repo definitions] **********
ok: [192.168.0.4]
TASK [../roles/add-elastic-repo : system update] *******************************
changed: [192.168.0.4]
PLAY [elk] *********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Installing Elasticsearch] ***********************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Replace default elasticsearch.yml] **************
changed: [192.168.0.4]
TASK [../roles/elasticsearch : service] ****************************************
changed: [192.168.0.4]
TASK [../roles/elasticsearch : create elasticsearch keystore] ******************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Set elasticsearch keystore permissions] *********
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Check if elasticsearch keystore is setup] *******
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Create bootstrap password for elastic user] *****
changed: [192.168.0.4]
TASK [../roles/elasticsearch : Restarting Elasticsearch] ***********************
changed: [192.168.0.4]
TASK [../roles/elasticsearch : Update elastic user password] *******************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Update Kibana user password] ********************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Update logstash_system user password] ***********
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Update beats_system user password] **************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Update APM system user password] ****************
ok: [192.168.0.4]
TASK [../roles/elasticsearch : Update remote_monitoring_user password] ********
ok: [192.168.0.4]
TASK [../roles/kibana : Installing Kibana with apt] ****************************
ok: [192.168.0.4]
TASK [../roles/kibana : Replacing default kibana.yml with updated file] ********
changed: [192.168.0.4]
TASK [../roles/kibana : Starting Kibana] ***************************************
ok: [192.168.0.4]
PLAY RECAP *********************************************************************
192.168.0.4 : ok=23 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
在实际的练习中,如果你想进行多次练习,请按照如下的步骤来操作:
- 删除 elasticsearch 安装: sudo dpkg -r elasticsearch
- 彻底删除 elasticsearch 安装: sudo apt-get --purge remove elasticsearch
- 删除目录:rm -rf /etc/elasticsearch 以及 /var/lib/elasticsearch
我们也可以按照同样的方法删除 Kibana:
- 删除 kibana:sudo dpkg -r elasticsearch
- 彻底删除 kibana:sudo apt-get --purge remove kibana
然后再重新作练习。
我们在 Ubuntu OS 的机器上运行如下的命令:
curl -u elastic:goodwitch http://localhost:9200
上面的命令显示:
我们在 MacOS 机器上访问 Kibana:
我们输入 Elasticsearch 的超级用户 elastic 的账号信息,并登陆:
在上面,我们成功地登陆了 Kibana。这说明我们的配置是成功的。
你可以在地址找到源码:https://github.com/liu-xiao-guo/elk-ansible
以上是关于Elastic:如何使用 Ansible自动化部署 Elastic Stack -Security的主要内容,如果未能解决你的问题,请参考以下文章
使用Ansible进行项目的自动部署(TomcatWeblogic)
Ansible 学习总结(10)—— 自动化部署工具如何选?Saltstack Or Ansible
Ansible 学习总结(10)—— 自动化部署工具如何选?Saltstack Or Ansible