logstash 输出带有序列号的弹性搜索索引

Posted 2023-03-22

【中文标题】logstash 输出带有序列号的弹性搜索索引

【英文标题】：logstash output elasticsearch index with sequence number

【发布时间】：2022-01-18 07:41:01

【问题描述】：

我正在使用带有 Logstash 7.10 的 AWS Elastic Search（版本 7.10）。目的是将内容从 logstash 发送到弹性搜索，并使用策略在特定大小或时间后翻转索引。

policy: 
    "policy_id": "Rollover_Policy",
    "description": "roller index",
    "last_updated_time": 1634910129219,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        
            "name": "hot",
            "actions": [
                
                    "rollover": 
                        "min_size": "1mb"
                    
                
            ],
            "transitions": [
                
                    "state_name": "warm"
                
            ]
        ,
        
            "name": "warm",
            "actions": [
                
                    "replica_count": 
                        "number_of_replicas": 1
                    
                
            ],
            "transitions": [
                
                    "state_name": "delete",
                    "conditions": 
                        "min_index_age": "1h"
                    
                
            ]
        ,
        
            "name": "delete",
            "actions": [
                
                    "delete": 
                
            ],
            "transitions": []
        
    ],
    "ism_template": [
        
            "index_patterns": [
                "products*"
            ],
            "priority": 100,
            "last_updated_time": 1634910129219
        
    ]

虽然我尝试在 logstash 输出插件中将 ilm_enabled 设置为 true，但它无法连接弹性搜索 xpack API。

注意：AWS 弹性搜索不支持 xpack 和 ILM。

elasticsearch   
        hosts => "$elasticsearch_endpoint"
        user => "$elasticsearch_user"
        password => "$elasticsearch_password"
        ilm_enabled => true
        ilm_rollover_alias => "products"
        ilm_pattern => "now/d-000001"
        ilm_policy => "Rollover_Policy"

所以我已将 ilm_enabled 标志更改为 false 并尝试以下选项

elasticsearch 
        hosts => "$elasticsearch_endpoint"
        user => "$elasticsearch_user"
        password => "$elasticsearch_password"
        ilm_enabled => false
        index => "products-%+YYYY.MM.dd-000001"

现在的问题是，即使在翻转之后，logstash 仍然将文档发送到 001 索引而不是新索引。如果我不在索引名称中给出 -000001，则翻转失败。

【参考方案1】：

在弹性中使用以下 REST 请求创建索引。由于索引名称具有日期模式，翻转将创建具有当前日期的新索引。

PUT %3Cproducts-%7Bnow%2Fd%7D-000001%3E

  "settings":
    "number_of_shards":1,
    "number_of_replicas":1
  ,
  "aliases": 
    "products":  
      "is_write_index": true
    
  

为索引模式创建一个模板以及翻转别名

PUT _index_template/products_logs

  "index_patterns": [
    "products*"
  ],
  "template": 
    "settings": 
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "opendistro": 
        "index_state_management": 
          "rollover_alias": "products"
        
      
    
  

在 logstash 输出插件中提供以下详细信息以将数据发送到弹性搜索

elasticsearch   
        hosts => "$elasticsearch_endpoint"
        user => "$elasticsearch_user"
        password => "$elasticsearch_password"
        ilm_enabled => false 
        index => "products"

注意：索引名代表索引的别名。