Linux | ELK 8.2搭建ELKB集群Ⅰ—— 实验环境说明和搭建Elasticsearch集群

Posted 2023-03-07 NOWSHUT

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Linux | ELK 8.2搭建ELKB集群Ⅰ—— 实验环境说明和搭建Elasticsearch集群相关的知识，希望对你有一定的参考价值。

1. 实验环境

1.1 实验工具

VMware® Workstation 16 Pro

1.2 操作系统

CentOS 7.9.2009 （Linux）

1.3 架构版本、IP地址规划与虚拟机配置要求

开源软件	虚拟机IP地址	版本	CPU与内存
Elasticsearch	192.168.100.31，192.168.100.32	8.2.1	大于1cpu, 大于1G内存
Logstash	192.168.100.33	8.2.0	大于1cpu, 大于1G内存
Kibana	192.168.100.33	8.2.1	大于1cpu, 大于1G内存
Filebeat	192.168.100.31-33	8.2.1	大于1cpu, 大于1G内存

1.4 拓扑图

1.5 其他要求

所有虚拟机需要连接互联网，虚拟机网卡模式设置为NAT或桥接模式

2. 实验步骤

以下操作需要在三台虚拟机上同时进行

关闭防火墙

systemctl stop firewall
systemctl disabled firewall

将SELinux设置为disabled

vim /etc/selinux/config 

SELINUX=disabled

同步服务器时间

yum install ntp            #安装ntp服务
systemctl start ntpd       #启动ntp
systemctl enable ntpd      #设置开机自启
date                       #三台服务器的时间一致即可

2.1 安装Elasticsearch（单节点）

官方安装包下载地址：https://www.elastic.co/cn/downloads/elasticsearch

（1）检查系统jdk版本

rpm -qa | grep openjdk
java -version

如果系统没有java环境，需要自行安装。

yum install java

再次检查jdk环境

（2）下载elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.2.2-x86_64.rpm

（3）安装elasticsearch

rpm -ivh elasticsearch-8.2.2-x86_64.rpm

警告：elasticsearch-8.2.2-x86_64.rpm: 头V4 RSA/SHA512 Signature, 密钥 ID d88e42b4: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:elasticsearch-0:8.2.2-1          ################################# [100%]
--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : GjKOXtfn5q1ZlHq7dM2K    #内置超级用户密码

If this node should join an existing cluster, you can reconfigure this with           #加入现有集群的命令
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with                          #重置es内置超级用户的密码
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with                           #为 Kibana 实例生成一个注册令牌
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with                #为 Elasticsearch 节点生成一个注册令牌
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

使用rpm包安装的elasticsearch其配置目录在/etc/elasticsearch；安装目录在/usr/share/elasticsearch

小技巧：通过rpm -qc命令查看elasticsearch的配置文件路径

# rpm -qc elasticsearch-8.2.2-1.x86_64
/etc/elasticsearch/elasticsearch-plugins.example.yml
/etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/jvm.options
/etc/elasticsearch/log4j2.properties
/etc/elasticsearch/role_mapping.yml
/etc/elasticsearch/roles.yml
/etc/elasticsearch/users
/etc/elasticsearch/users_roles
/etc/sysconfig/elasticsearch
/usr/lib/sysctl.d/elasticsearch.conf
/usr/lib/systemd/system/elasticsearch.service

（4）配置elasticsearch

vim /etc/elasticsearch/elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elk-cluster                    #自定义集群名
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1                           #自定义节点名
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch            #elasticsearch数据存放路径
#
# Path to log files:
#
path.logs: /var/log/elasticsearch            #elasticsearch日志存放路径
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0            #设置能访问elasticsearch的IP地址，0.0.0.0表示所有IP都能访问，监听所有IP
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200                 #设置elasticsearch数据传输端口号，即监听端口，默认为9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 04-06-2022 20:18:05
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true                             #elasticsearch v7以后自动开启安全模式

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["vms31.rhce.cc"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

修改完后使用cat命令查看设置

# cat /etc/elasticsearch/elasticsearch.yml | grep -Ev "#|^$" 
cluster.name: elk-cluster
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
cluster.initial_master_nodes: ["vms31.rhce.cc"]
http.host: 0.0.0.0

（5）启动与查看服务

启动服务

systemctl start elasticsearch.service 
systemctl enable elasticsearch.service

查看启动端口

# netstat -ntlup | grep java
tcp6       0      0 :::9200                 :::*                    LISTEN      24625/java          
tcp6       0      0 :::9300                 :::*                    LISTEN      24625/java

其中9200是数据传输端口，9300示集群通信端口。

（6）访问elasticsearch

使用curl命令访问

# curl -u elastic:GjKOXtfn5q1ZlHq7dM2K https://192.168.100.31:9200/ --insecure

  "name" : "vms31.rhce.cc",
  "cluster_name" : "elk-cluster",
  "cluster_uuid" : "4IoxZ9U5T_-7T26soNLm8A",
  "version" : 
    "number" : "8.2.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "9876968ef3c745186b94fdabd4483e01499224ef",
    "build_date" : "2022-05-25T15:47:06.259735307Z",
    "build_snapshot" : false,
    "lucene_version" : "9.1.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  ,
  "tagline" : "You Know, for Search"

使用浏览器访问：https://192.168.100.31:9200/

2.2 搭建Elasticsearch集群

为了安装elasticsearch集群，我们将

192.168.100.31节点设置为master
192.168.100.32和192.168.100.33节点视为node

（1）安装elasticsearch

根据2.1的（1）——（3），在另外两台服务器192.168.100.32和192.168.100.33上安装elasticsearch

（2）配置elasticsearch集群

设置节点192.168.100.31

# cat /etc/elasticsearch/elasticsearch.yml | grep -v "#"
cluster.name: elk-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: 
  - 192.168.100.31:9300
  - 192.168.100.32:9300
cluster.initial_master_nodes: ["node-1", "node-2"]


xpack.security.enabled: false

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

http.host: 0.0.0.0

设置节点192.168.100.32

# cat /etc/elasticsearch/elasticsearch.yml | grep -v "#"
cluster.name: elk-cluster
node.name: node-2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.100.31:9300", "192.168.100.32:9300"]
cluster.initial_master_nodes: ["node-1", "node-2"]


xpack.security.enabled: false

xpack.security.enrollment.enabled: true

xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

http.host: 0.0.0.0

总结

初始化集群时，所有节点的

集群名cluster.name要一致，
集群IP地址discovery.seed_hosts要一致，

（3）重启elasticsearch服务

systemctl restart elasticsearch

（4）查看es集群信息

# curl http://192.168.100.31:9200/_cluster/health?pretty

  "cluster_name" : "elk-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 2,
  "active_shards" : 4,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0


# curl http://192.168.100.31:9200/_nodes/process?pretty

  "_nodes" : 
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  ,
  "cluster_name" : "elk-cluster",
  "nodes" : 
    "8bB4P1EET2mjhecE4fez9Q" : 
      "name" : "node-2",
      "transport_address" : "192.168.100.32:9300",
      "host" : "192.168.100.32",
      "ip" : "192.168.100.32",
      "version" : "8.2.2",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "9876968ef3c745186b94fdabd4483e01499224ef",
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "attributes" : 
        "ml.machine_memory" : "4122771456",
        "ml.max_jvm_size" : "2063597568",
        "xpack.installed" : "true"
      ,
      "process" : 
        "refresh_interval_in_millis" : 1000,
        "id" : 52915,
        "mlockall" : false
      
    ,
    "c69H-_ToSLOsbiiIZnY6QA" : 
      "name" : "node-1",
      "transport_address" : "192.168.100.31:9300",
      "host" : "192.168.100.31",
      "ip" : "192.168.100.31",
      "version" : "8.2.2",
      "build_flavor" : "default",
      "build_type" : "rpm",
      "build_hash" : "9876968ef3c745186b94fdabd4483e01499224ef",
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ],
      "attributes" : 
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "2063597568",
        "ml.machine_memory" : "4122771456"
      ,
      "process" : 
        "refresh_interval_in_millis" : 1000,
        "id" : 38585,
        "mlockall" : false

参考资料

Elasticsearch介绍：Elasticsearch 是什么？
Elasticsearch文档：Elasticsearch Guide
Logstash文档：Logstash Reference
Kibana文档：Kibana Guide
Filebeat文档：Filebeat Reference

关联博文

由于篇幅原因，关于搭建ELKB集群其他内容请查阅：
安装 Logstash 和 Kibana
安装 Filebeat和问题与解决方案

以上是关于Linux | ELK 8.2搭建ELKB集群Ⅰ—— 实验环境说明和搭建Elasticsearch集群的主要内容，如果未能解决你的问题，请参考以下文章

ELKB5.2.2集群环境部署及优化终极文档

ELK 集群搭建总结

一步一下搭建ELK集群

elk6.3 centos集群搭建 head插件安装

Linux下ELK环境搭建

ELK集群搭建简略记录