如何将 fluentd 连接到 opendistro 以进行弹性搜索

Posted

技术标签:

【中文标题】如何将 fluentd 连接到 opendistro 以进行弹性搜索【英文标题】：How to connect fluentd to opendistro for elasticsearch 【发布时间】：2021-12-18 22:01:06 【问题描述】：

我已经创建了一个 docker 文件：

FROM fluentd:v1.14.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.1.2"]
USER fluent

我在撰写文件中创建了一个容器：

version: '3'
services:
  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    restart: always
    container_name: fluentd
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    networks:
      - network
  elasticsearch:
    image: docker pull amazon/opendistro-for-elasticsearch:latest
    container_name: elasticsearch
    environment:
      - cluster.name=elasticsearch
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - elasticsearch:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - network
  kibana:
    image: docker pull amazon/opendistro-for-elasticsearch-kibana:latest
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://elasticsearch:9200
      ELASTICSEARCH_HOSTS: https://elasticsearch:9200
    networks:
      - network 
volumes:
  elasticsearch:

networks:
  network:
    driver: bridge
    name: network

这是我的 fluent.conf

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    user admin
    password admin
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    scheme https
    ssl_verify false
  </store>

  <store>
    @type stdout
  </store>
</match>

由于某种原因，elasticsearch 和 kibana 启动时没有问题，但 fluentd 无法连接。这是我得到的错误

 *

2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.2/lib/fluent/plugin/elasticsearch_index_template.rb:52:in `retry_operate'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.2/lib/fluent/plugin/out_elasticsearch.rb:487:in `handle_last_seen_es_major_version'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.1.2/lib/fluent/plugin/out_elasticsearch.rb:339:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:110:in `block in configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `each'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/multi_output.rb:99:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin/out_copy.rb:39:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/plugin.rb:187:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:132:in `add_match'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:74:in `block in configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `each'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/agent.rb:64:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/root_agent.rb:149:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:105:in `configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/engine.rb:80:in `run_configure'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:716:in `block in run_worker'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:968:in `main_process'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/supervisor.rb:708:in `run_worker'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.0/bin/fluentd:15:in `<top (required)>'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/bin/fluentd:23:in `load'
    fluentd          |   2021-11-04 16:47:35 +0000 [error]: #0 /usr/bin/fluentd:23:in `<main>'
    fluentd          | 2021-11-04 16:47:35 +0000 [info]: Worker 0 finished unexpectedly with status 1
    fluentd          | 2021-11-04 16:47:43 +0000 [info]: adding match pattern="*.**" type="copy"
    fluentd          | 2021-11-04 16:47:45 +0000 [error]: #0 unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not a supported distribution of Elastic

我不确定用于 fluentd 连接到 elasticsearch 的插件是否也有资格连接到 OpenDistro 以进行 elasticsearch。如果需要任何其他信息，请询问。 任何帮助将不胜感激。

【问题讨论】：

【参考方案1】：

使用稳定的最新版本插件elasticsearch 和fluent-plugin-elasticsearch。 以我工作的 Dockerfile 为例：

FROM fluent/fluentd:v1.12.0-debian-1.0 用户根 运行 ["gem", "install", "elasticsearch", "--no-document", "--version", "7.13.3"] 运行 ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.5"] 用户流利

【讨论】：

尝试使用 7.10 的 elasticsearch 和 5.0.0 的插件，一切正常。但建议将来参考使用 fluent-bit 而不是 fluentd，因为它带有一个用于连接到 elasticsearch 的内置插件，它更小，甚至更快。【参考方案2】：

问题在于该插件适用于较低版本的 elasticsearch，在重构我的 docker 文件后，一切都像一个魅力：

FROM fluentd:v1.14.0-debian-1.0
USER root
RUN gem install elasticsearch -v 7.13.3 \
&& gem install fluent-plugin-elasticsearch --no-document
USER fluent

【讨论】：

以上是关于如何将 fluentd 连接到 opendistro 以进行弹性搜索的主要内容，如果未能解决你的问题，请参考以下文章

如何将 HTML 页面连接到 MongoDB？

将 Auth0 连接到我的后端后，如何将其连接到我的 NextJS 前端？

如何将设备连接到 Eclipse？

Android 10 / API 29：如何将手机连接到配置的网络？

如何将流星连接到现有后端？

如何将标签栏项目连接到操作？