Elasticsearch日志收集

Posted 2021-01-19 ryansunyu

Install pip if necessary

curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" python get-pip.py

Install Curator for Elasticsearch

Elasticsearch Curator helps you curate, or manage, your Elasticsearch indices and snapshots by:

• Obtaining the full list of indices (or snapshots) from the cluster, as the actionable list
• Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list as needed.
• Perform various actions on the items which remain in the actionable list.

 

pip install elasticsearch-curator pip install click==6.7

 

Configure curator

mkdir -p /var/log/elastic touch /var/log/elastic/curator.log mkdir ~/.curator vi ~/.curator/curator.yml

curator.yml

# Remember, leave a key empty if there is no value. None will be a string, ## not a Python "NoneType" client: hosts: [Elasticsearch Server IP] port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False   logging: loglevel: INFO logfile: /var/log/elastic/curator.log logformat: default blacklist: [‘elasticsearch‘, ‘urllib3‘]

Have a test, now you can get the indices list
curator_cli show_indices

 

Create repository

Configure elasticseach.yml default in /etc/elasticsearch/elasticsearch.yml

elasticsearch.yml

path.repo:  /u01/elasticsearch/backup http.max_header_size: 16kb

Restart elasticsearch service (service elasticsearch restart) to make the configurations work.

Create repository elasticsearch. Ensure location points to a valid path which is configured in path.repo, accesable from all nodes.

curl -XPUT http://localhost:9200/_snapshot/es_backup -H "Content-Type: application/json" -d @repository.json

repository.json

{    "type": "fs",    "settings": {       "compress": true,       "location": "/u01/elasticsearch/backup"    } }

 

Have a test

curl -XGET ‘localhost:9200/_snapshot/_all?pretty=true‘

Create curator yaml action files

daily_backup.yml

Customize the snapshot name in name option
action 1: backup all indices before today to repository elasticsearch with specified snapshot name
action 2: delete indices older than 185 days

daily_backup.yml

--- actions:   1:     action: snapshot     description: >-       Snapshot selected all indices to repository ‘elasticsearch‘ with the snapshot name     options:       repository: es_backup       name: ‘<c4cert-{now/d-1d}>‘       wait_for_completion: True       max_wait: 4800       wait_interval: 30     filters:     - filtertype: age       source: name       direction: older       unit: days       unit_count: 1       timestring: "%Y.%m.%d"       2:     action: delete_indices     description: >-       Delete indices which is older than 185 days     filters:     - filtertype: age       source: name       direction: older       unit: days       unit_count: 185       timestring: "%Y.%m.%d"

 

del_snapshot.yml
action 1: Delete snapshots from repository elasticsearch which is older than 185 days

del_snapshot.yml

---   actions:   1:     action: delete_snapshots     description: >-       Delete snapshots from repository which is older than 185 days     options:       repository: es_backup       retry_interval: 120       retry_count: 3     filters:     - filtertype: age       source: creation_date       direction: older       unit: days       unit_count: 185

 

restore.yml
action 1: Restore all indices in the most recent snapshot with state SUCCESS.

restore.yml

---   actions:   1:     action: restore     description: >-       Restore all indices in the most recent snapshot with state SUCCESS.  Wait       for the restore to complete before continuing.  Do not skip the repository       filesystem access check.  Use the other options to define the index/shard       settings for the restore.     options:       repository: es_backup       # If name is blank, the most recent snapshot by age will be selected       name:       # If indices is blank, all indices in the snapshot will be restored       indices:       wait_for_completion: True       max_wait: 3600       wait_interval: 10     filters:     - filtertype: state       state: SUCCESS

 

Note: use --dry-run option to verify your action without any change. Find the dry run results in log path.
Curator --dry-run daily_backup.yml

 

Shell script and crontab

run.sh

#!/bin/sh curator /u01/curator/del_snapshot.yml curator /u01/curator/daily_backup.yml

crontab -e

Here configured the job run on every 3 AM

crontab

0 3 * * * /bin/sh /u01/curator/run.sh

 

Restore

Curator restore.yml

Tested OK in CERT env.

 

Some useful API 

# get all repositories curl -XGET ‘localhost:9200/_snapshot/_all?pretty=true‘   # delete repository curl -XDELETE ‘localhost:9200/_snapshot/es-snapshot?pretty=true‘   # show snapshots curator_cli show_snapshots --repository es_backup   # show indices curator_cli show_indices

以上是关于Elasticsearch日志收集的主要内容，如果未能解决你的问题，请参考以下文章

filebeat+logstash+elasticsearch收集haproxy日志

filebeat+elasticsearch+logstash+kibana收集系统日志(docker)

Tomcat容器日志收集方案fluentd+elasticsearch+kilbana

搭建Graylog2集群（基于ElasticSearch的日志收集分析平台）

Fluentd+Elasticsearch+kibana 日志收集部署实战

Elasticsearch日志收集