Dockerized elasticsearch and fscrawler: failed to create elasticsearch client, disable crawler... Co

Posted 2023-03-22

【中文标题】Dockerized elasticsearch and fscrawler: failed to create elasticsearch client, disable crawler... Connection denied

【英文标题】：Dockerized elasticsearch and fscrawler: failed to create elasticsearch client, disabling crawler… Connection refused

【发布时间】：2020-11-30 14:00:28

【问题描述】：

我在尝试将 Dockerized fscrawler 连接到 Dockerized elasticsearch 时收到以下错误：

[f.p.e.c.f.c.ElasticsearchClientManager] 创建失败 elasticsearch 客户端，禁用爬虫... [f.p.e.c.f.FsCrawler] 致命 运行爬虫时收到错误：[连接被拒绝]

【参考方案1】：

当 fscrawler 第一次运行（即docker-compose run fscrawler）时，它会使用以下默认设置创建/config/fscrawer_job/_settings.yml：

elasticsearch:
  nodes:
  - url: "http://127.0.0.1:9200"

这将导致 fscrawler 尝试连接到 localhost（即 127.0.0.1）。但是，当 fscrawler 位于 docker 容器中时，这将失败，因为它正在尝试与 CONTAINER 的本地主机连接。这在我的情况下特别令人困惑，因为 elasticsearch 可以作为 localhost 访问，但在我的物理计算机的 localhost 上（而不是容器的 localhost）。更改 url 允许 fscrawler 连接到 elasticsearch 实际所在的网络地址。

elasticsearch:
  nodes:
  - url: "http://elasticsearch:9200"

我使用了以下 docker 镜像：https://hub.docker.com/r/toto1310/fscrawler

# FILE: docker-compose.yml

version: '2.2'
services:
  # FSCrawler 
  fscrawler:
    image: toto1310/fscrawler
    container_name: fscrawler
    volumes:
      - $PWD/config:/root/.fscrawler
      - $PWD/data:/tmp/es
    networks: 
      - esnet
    command: fscrawler job_name

  # Elasticsearch Cluster
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: elasticsearch
    environment:
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch2
      - cluster.initial_master_nodes=elasticsearch,elasticsearch2
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: elasticsearch2
    environment:
      - node.name=elasticsearch2
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch,elasticsearch2
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata02:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata01:
    driver: local
  esdata02:
    driver: local

networks:
  esnet:

跑docker-compose up elasticsearch elasticsearch2 调出弹性搜索节点。 跑docker-compose run fscrawler创建_settings.yml 编辑_settings.yml到

elasticsearch:
  nodes:
  - url: "http://elasticsearch:9200"

启动 fscrawler docker-compose up fscrawler