elasticsearch的安装部署

Posted 2020-11-26 yatou-blog

环境部署

1.服务器准备：centos7.4系统，Jdk1.8 cat /etc/redhat-release。Es6.x

如果是自带的openjdk需要先卸载：

rpm -qa|grep java

rpm -e –nodeps *

2.ElasticSearch安装：我采用的版本为6.6.0 https://elasticsearch.cn/download/

（1）地址：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.0.tar.gz 

（2）解压：tar -zxvf elasticsearch/elasticsearch-6.6.0.tar.gz

 (3)修改系统配置：

• 设置内核参数 ：vim /etc/sysctl.conf

　　　　　添加如下内容:
                fs.file-max=65536
                vm.max_map_count=262144
            sysctl -p 刷新下配置，

　　　　sysctl -a查看是否生效  ，

如果不成功的(启动es还是失败,不是所有人都碰得到，好像是在7.6碰到了)：

　　　　rm -f /sbin/modprobe ?

　　　　ln -s /bin/true /sbin/modprobe
　　　　rm -f /sbin/sysctl 

　　　　?ln -s /bin/true /sbin/sysctl

• 设置资源参数
               vi /etc/security/limits.conf
               # 添加一下内容：
               * soft nofile 65536
               * hard nofile 131072
               * soft nproc  2048
              * hard nproc  4096

• 修改进程数
               vi /etc/security/limits.d/20-nproc.conf
                  *          soft    nproc     4096
  配置完成后 要关掉链接窗口，重新打开一个
  不能用root用户启动

• 修改es配置文件： vim elasticsearch安装路径/config/elasticsearch.yml
  # 配置es的集群名称, es会自动发现在同一网段下的es,如果在同一网段下有多个集群,就可以用这个属性来区分不同的集群?
  cluster.name: bi-cluster
  # 节点名称
  node.name: node-master
  # 存储数据的目录
  path.data: /home/elasticsearch/data
  # 存储日志的目录
  path.logs: /home/elasticsearch/logs
  # 设置绑定的ip地址还有其它节点和该节点交互的ip地址
  network.host: 0.0.0.0
  # 指定http端口,你使用head?kopf等相关插件使用的端口
  http.port: 9200
  # 设置节点间交互的tcp端口,默认是9300
  transport.tcp.port: 9300
  #设置集群中master节点的初始列表,可以通过这些节点来自动发现新加入集群的节点?
  discovery.zen.ping.unicast.hosts: ["10.108.4.203:9300", "10.108.4.204:9300", "10.108.4.205:9300"]
  #如果没有这种设置,遭受网络故障的集群就有可能将集群分成两个独立的集群 - 分裂的大脑 - 这将导致数据丢失
  discovery.zen.minimum_master_nodes: 3
  http.cors.enabled: true        #es5.x版本以上需要，head访问
  http.cors.allow-origin: "*"
  bootstrap.memory_lock: false    #某些系统需要 是因为centos6.x操作系统不支持SecComp，而elasticsearch 5.5.2默认 bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。
  我的配置如下：

  # ======================== Elasticsearch Configuration ====================
  =====
  #
  # NOTE: Elasticsearch comes with reasonable defaults for most settings.
  #       Before you set out to tweak and tune the configuration, make sure y
  ou
  #       understand what are you trying to accomplish and the consequences.
  #
  # The primary way of configuring a node is via this file. This template lis
  ts
  # the most important settings you may want to configure for a production cl
  uster.
  #
  # Please consult the documentation for further information on configuration
   options:
  # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
  #
  # ---------------------------------- Cluster ------------------------------
  -----
  #
  # Use a descriptive name for your cluster:
  #
  #cluster.name: my-application
  #
  # ------------------------------------ Node -------------------------------
  -----
  #
  # Use a descriptive name for the node:
  #
  #node.name: node-1
  #
  # Add custom attributes to the node:
  #
  #node.attr.rack: r1
  #
  # ----------------------------------- Paths -------------------------------
  -----
  #
  # Path to directory where to store the data (separate multiple locations by
   comma):
  #
  #path.data: /path/to/data
  #
  # Path to log files:
  #
  #path.logs: /path/to/logs
  #
  # ----------------------------------- Memory ------------------------------
  -----
  #
  # Lock the memory on startup:
  #
  #bootstrap.memory_lock: true
  #
  # Make sure that the heap size is set to about half the memory available
  # on the system and that the owner of the process is allowed to use this
  # limit.
  #
  # Elasticsearch performs poorly when the system is swapping the memory.
  #
  # ---------------------------------- Network ------------------------------
  -----
  #
  # Set the bind address to a specific IP (IPv4 or IPv6):
  #
  #network.host: 192.168.0.1
  #
  # Set a custom port for HTTP:
  #
  #http.port: 9200
  #
  # For more information, consult the network module documentation.
  #
  # --------------------------------- Discovery -----------------------------
  -----
  #
  # Pass an initial list of hosts to perform discovery when new node is start
  ed:
  # The default list of hosts is ["127.0.0.1", "[::1]"]
  #
  #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
  #
  # Prevent the "split brain" by configuring the majority of nodes (total num
  ber of master-eligible nodes / 2 + 1):
  #
  #discovery.zen.minimum_master_nodes: 
  #
  # For more information, consult the zen discovery module documentation.
  #
  # ---------------------------------- Gateway ------------------------------
  -----
  #
  # Block initial recovery after a full cluster restart until N nodes are sta
  rted:
  #
  #gateway.recover_after_nodes: 3
  #
  # For more information, consult the gateway module documentation.
  #
  # ---------------------------------- Various ------------------------------
  -----
  #
  # Require explicit names when deleting indices:
  #
  #action.destructive_requires_name: true
  cluster.name: my-es
  node.name: node-128
  network.host: 0.0.0.0
  http.port: 19200
  transport.tcp.port: 19300
  #discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"]
  #discovery.zen.minimum_master_nodes: 3
  http.cors.enabled: true        
  http.cors.allow-origin: "*"
  bootstrap.memory_lock: false    
  bootstrap.system_call_filter: false
  

  　　

• 启动es：./bin/elasticsearch 后台启动加 –d参数

• 打开浏览器访问 http://你的ip:19200/查看是否能够正常访问,看到以下界面表示启动ok

  

   

   

    

如果在外网访问失败的话，请按照以下步骤执行：

1. 检查你的elasticsearch.yml配置文件的以下三处是否配置正确

   network.host: 0.0.0.0

   bootstrap.memory_lock: false    
   bootstrap.system_call_filter: false

2. 检查防火墙是否关闭（需要关闭防火墙外网才可以访问）
   

   开启、重启、关闭、firewalld.service服务

   # 开启
   service firewalld start
   # 重启
   service firewalld restart
   # 关闭
   service firewalld stop
   查看防火墙规则：

   firewall-cmd --list-all
   查询、开放、关闭端口：

   # 查询端口是否开放
   firewall-cmd --query-port=8080/tcp
   # 开放80端口
   firewall-cmd --permanent --add-port=80/tcp
   # 移除端口
   firewall-cmd --permanent --remove-port=8080/tcp
   #重启防火墙(修改配置后要重启防火墙)
   firewall-cmd --reload

   执行以上步骤之后就可以在外网访问了。以下是我遇到的启动错误的问题以及解决办法：
3. （1）max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
   
   　每个进程最大同时打开文件数太小，可通过下面2个命令查看当前数量

4. ulimit -Hn
   ulimit -Sn

   　　修改/etc/security/limits.conf文件，增加配置，用户退出后重新登录生效

   *               soft    nofile          65536
   *               hard    nofile          65536

   

    

   （2）max number of threads [3818] for user [es] is too low, increase to at least [4096]

   　　问题同上，最大线程个数太低。修改配置文件/etc/security/limits.conf（和问题1是一个文件），增加配置

   *               soft    nproc           4096
   *               hard    nproc           4096

   　　可通过命令查看

   ulimit -Hu
   ulimit -Su

   

   修改后的文件：

   

    如果更改后发现ulimit -Su 的结果没有变化的话，此时需要在这个文件中添加如下信息：(admin是我自己的用户名称，可以根据自己的实际用户更改)

   admin - nproc 4096

   

    

    

    

5. max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

   　　修改/etc/sysctl.conf文件，增加配置vm.max_map_count=262144

   vi /etc/sysctl.conf
   sysctl -p

   　　执行命令sysctl -p生效

   

    4、Exception in thread "main" java.nio.file.AccessDeniedException: /usr/local/elasticsearch/elasticsearch-6.2.2-1/config/jvm.options

   　　elasticsearch用户没有该文件夹的权限，执行命令

   chown -R es:es /usr/local/elasticsearch/

安装kibana：

1、将下载的kibana安装包解压： tar -zxvf kibana-6.6.0-linux-x86_64.tar.gz

2、更改kibana的配置

# Kibana is served by a back end server. This setting specifies the port to
 use.
server.port: 15601

# Specifies the address to which the Kibana server will bind. IP addresses 
and host names are both valid values.
# The default is ‘localhost‘, which usually means remote machines will not 
be able to connect.
# To allow connections from remote users, set this parameter to a non-loopb
ack address.
#server.host: "localhost"
server.host: "0.0.0.0"


# Enables you to specify a path to mount Kibana at if you are running behin
d a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remo
ve the basePath
# from requests it receives, and to prevent a deprecation warning at startu
p.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse prox
y.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server‘s name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.hosts: ["http://localhost:19200"]


# When this setting‘s value is true Kibana uses the hostname specified in t
he server.host
# setting. When the value of this setting is false, Kibana uses the hostnam
e of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizat
ions and
# dashboards. Kibana creates a new index if the index doesn‘t already exist
.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these setti
ngs provide
# the username and password that the Kibana server uses to perform maintena
nce on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elast
icsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files
, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to
 the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificat
e and key files.
# These files validate that your Elasticsearch backend uses the same key fi
les.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for t
he certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting‘s valu
e to ‘none‘.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defau
lts to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticse
arch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no*
 client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom header
s cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhi
telist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards.
 Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before r
etrying.
#elasticsearch.startupTimeout: 5000

[admin@localhost kibana-6.6.0-linux-x86_64]$ cat config/kibana.yml         
# Kibana is served by a back end server. This setting specifies the port to
 use.                                                                      
server.port: 15601                                                         
                                                                           
# Specifies the address to which the Kibana server will bind. IP addresses 
and host names are both valid values.                                      
# The default is ‘localhost‘, which usually means remote machines will not 
be able to connect.                                                        
# To allow connections from remote users, set this parameter to a non-loopb
ack address.                                                               
#server.host: "localhost"                                                  
server.host: "0.0.0.0"                                                     
                                                                           
                                                                           
# Enables you to specify a path to mount Kibana at if you are running behin
d a proxy.                                                                 
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remo
ve the basePath                                                            
# from requests it receives, and to prevent a deprecation warning at startu
p.                                                                         
# This setting cannot end in a slash.                                      
#server.basePath: ""                                                       
                                                                           
# Specifies whether Kibana should rewrite requests that are prefixed with  
# `server.basePath` or require that they are rewritten by your reverse prox
y.                                                                         
# This setting was effectively always `false` before Kibana 6.3 and will   
# default to `true` starting in Kibana 7.0.                                
#server.rewriteBasePath: false                                             

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server‘s name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.hosts: ["http://localhost:19200"]


# When this setting‘s value is true Kibana uses the hostname specified in t
he server.host
# setting. When the value of this setting is false, Kibana uses the hostnam
e of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizat
ions and
# dashboards. Kibana creates a new index if the index doesn‘t already exist
.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these setti
ngs provide
# the username and password that the Kibana server uses to perform maintena
nce on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elast
icsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "user"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files
, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to
 the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificat
e and key files.
# These files validate that your Elasticsearch backend uses the same key fi
les.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for t
he certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting‘s valu
e to ‘none‘.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defau
lts to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticse
arch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no*
 client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom header
s cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhi
telist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards.
 Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before r
etrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output othe
r than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system
 usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number
 formats.
#i18n.locale: "en"

3、启动kibana:  ./bin/kibana

 

 4、外网通过ip地址访问：

 

 

注意两个点：
（1）修改启动的es链接地址
（2）修改地址绑定不然只有本地才能访问

（3）kibana的版本和elasticsearch的版本必须一致，比如都是6.6