ELK

Posted cjwnb

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了ELK相关的知识，希望对你有一定的参考价值。

节点安装

172.16.240.20   es kibana
172.16.240.30    logstash


安装jdk1.8

2个节点都安装


  • 安装
cd /usr/local/src/

ls
  jdk-8u231-linux-x64.tar.gz
  
tar -zxf jdk-8u231-linux-x64.tar.gz 
ls
  jdk1.8.0_231  jdk-8u231-linux-x64.tar.gz
  
mv jdk1.8.0_231/ /usr/local/

/usr/local/jdk1.8.0_231/bin/java -version
  java version "1.8.0_231"
  Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
  Java HotSpot(TM) 64-Bit Server VM (build 25.231-b11, mixed mode)


  • 配置环境变量
vim  /etc/profile
  jave_home=/usr/local/jdk1.8.0_231/bin
  PATH=$PATH:$HOME/bin:$jave_home


安装kibana

172.16.240.20节点安装

cd /usr/local/src/
tar -zxf kibana-6.6.0-linux-x86_64.tar.gz
mv kibana-6.6.0-linux-x86_64 /usr/local/kibana-6.6.0
vim /usr/local/kibana-6.6.0/config/kibana.yml
  server.port: 5601
  server.host: "0.0.0.0"
  
/usr/local/kibana-6.6.0/bin/kibana &     # 前台启动kibana服务
nohup /usr/local/kibana-6.6.0/bin/kibana >> /tmp/kibana.log 2>/tmp/kibana.log &


Kibana通过nginx实现认证

  • Kibana监听在127.0.0.1
  • 部署Nginx，使用Nginx来转发
yum install -y lrzsz wget gcc gcc-c++ make pcre pcre-devel zlib zlib-devel
cd /usr/local/src/
tar -zxf nginx-1.14.2.tar.gz 
cd nginx-1.14.2/
./configure --prefix=/usr/local/nginx && make && make install
vim ~/.bash_profile
  nginx_path=/usr/local/nginx/sbin/
  PATH=$PATH:$HOME/bin:$jave_home:$nginx_path


kibana端配置

vim /usr/local/kibana-6.6.0/config/kibana.yml
  server.port: 5601
  server.host: "127.0.0.1"


nginx通过添加白名单访问

>>>> 查看本机的vmnet8的ip地址
ip a |grep vmnet8 |awk 'NR==2{print $2}'|cut -d '/' -f1
172.16.240.1


vim /usr/local/nginx/conf/nginx.conf
  worker_processes  1;
  events {
      worker_connections  1024;
  }
  http {
      include       mime.types;
      default_type  application/octet-stream;
      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      sendfile        on;
      keepalive_timeout  65;
      server {
          listen       80;
          server_name  localhost;
          location / {
                  allow 127.0.0.1;
                  allow 172.16.240.1;
                  deny all;
                  proxy_pass http://127.0.0.1:5601;
          }
          error_page   500 502 503 504  /50x.html;
          location = /50x.html {
              root   html;
          }
      }
  }
  

>>> 重新加载nginx
/usr/local/nginx/sbin/nginx -s reload


nginx通过验证用户名和密码

vim /usr/local/nginx/conf/nginx.conf
  worker_processes  1;
  events {
      worker_connections  1024;
  }
  http {
      include       mime.types;
      default_type  application/octet-stream;
      log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                        '$status $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
      sendfile        on;
      keepalive_timeout  65;
      server {
          listen       80;
          server_name  localhost;
          location / {
                  auth_basic "elk auth";
                  auth_basic_user_file /usr/local/nginx/conf/htpasswd;
                  proxy_pass http://127.0.0.1:5601;
          }
          error_page   500 502 503 504  /50x.html;
          location = /50x.html {
              root   html;
          }
      }
  }

 printf "elk:$(openssl passwd -1 elkpassword)
" > /usr/local/nginx/conf/htpasswd
 
 /usr/local/nginx/sbin/nginx -s reload


elasticsearch的安装

cd /usr/local/src/
tar -zxf elasticsearch-6.6.0.tar.gz
mv elasticsearch-6.6.0 /usr/local/

vim /usr/local/elasticsearch-6.6.0/config/elasticsearch.yml
  path.data:  /usr/local/elasticsearch-6.6.0/data
  path.logs: /usr/local/elasticsearch-6.6.0/logs 
  network.host: 127.0.0.1
  http.port: 9200
  
vim /usr/local/elasticsearch-6.6.0/config/jvm.options
  -Xms128M
  -Xmx128M
  

>>>> elasticsearch不能通过root用户启动

useradd -s /sbin/nologin elk
chown -R elk:elk /usr/local/elasticsearch-6.6.0/
su - elk -s /bin/bash
/usr/local/elasticsearch-6.6.0/bin/elasticsearch -d


注意

Elasticsearch启动注意
Elasticsearch如果启动在127.0.0.1的话，可以启动成功
Elasticsearch如果要跨机器通讯，需要监听在真实网卡上
监听在真实网卡需要调整系统参数才能正常启动

Elasticsearch监听在非127.0.0.1
监听在0.0.0.0或者内网地址
以上两种监听都需要调整系统参数

ulimit -a
  core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 7827
  max locked memory       (kbytes, -l) 64
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 65536
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) 8192
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) 4096
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited

ES启动三个报错的处理
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [3829] for user [elk] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

最大文件打开数调整/etc/security/limits.conf
* - nofile 65536

最大打开进程数调整/etc/security/limits.d/20-nproc.conf
* - nproc 10240

内核参数调整
vm.max_map_count = 262144

Elasticsearch监听网卡建议
如果学习，建议监听在127.0.0.1
如果是云服务器的话，一定把9200和9300公网入口在安全组限制一下
自建机房的话，建议监听在内网网卡，监听在公网会被入侵


Elasticsearch操作

结构: 索引--->> 类型 --->>id

索引层操作

PUT /lyysb    创建索引 lyysb
GET _cat/indices?v    获取所有的索引
DELETE /lyysb    删除索引 lyysb


文档层操作

/索引/类型/文档(_id)

创建和插入数据

PUT /lyysb/users/1
{
  "name": "lyysb",
  "age": 38
}


根据类型和id查询数据

GET /lyysb/users/1


查询索引下的所有数据

GET /lyysb/_search?q=*


修改数据(PUT)

操作和创建数据都是一样的

如果所有字段都改变, 则这个过程就是创建

如果修改部分字段的值, 这个过程就是更新

PUT /weixinyu/users/2
{
  "name": "wxy",
  "age": 18
}

PUT /weixinyu/users/2
{
  "name": "wxy",
  "age": 128
}


删除数据

DELETE /weixinyu/users/2


修改数据(POST)

状态永远为update

GET /weixinyu/_search?q=*

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "weixinyu",
        "_type" : "users",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "name" : "wxyzaruan33",
          "age" : 1122
        }
      }
    ]
  }
}


POST /weixinyu/users/1
{
  "name": "wxyzarddduan33",
  "age": 112222
}


{
  "_index" : "weixinyu",
  "_type" : "users",
  "_id" : "1",
  "_version" : 6,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 5,
  "_primary_term" : 1
}


更改所有数据

PUT /index/type/1
{
  "name": "wxy",
  "age": 33
}

PUT /index/type/2
{
  "name": "chenjun",
  "age": 31
}

PUT /index/type/3
{
  "name": "sharen",
  "age": 22
}


批量将所有人的年龄改为111

POST /index/type/_update_by_query
{
  "script": {
    "source": "ctx._source['age']=111"
  },
  
  "query": {
    "match_all": {}
  }
}


增加一个字段

POST /index/type/_update_by_query
{
  "script": {
    "source": "ctx._source['city']='hangzhou'"
  },
  
  "query": {
    "match_all": {}
  }
}


logstash安装

节点172.16.240.30安装

cd /usr/local/src/
tar -zxf logstash-6.6.0.tar.gz 
mv logstash-6.6.0 /usr/local/
vim /usr/local/logstash-6.6.0/config/jvm.options 
  -Xms200M
  -Xmx200M
  
vim /usr/local/logstash-6.6.0/config/logstash.conf
  input{
    stdin{}
  }
  output{
    stdout{
      codec=>rubydebug
    }
  }

以上是关于ELK的主要内容，如果未能解决你的问题，请参考以下文章

ELK 启动报错

elk日志分析搭建使用记录

elk日志分析搭建使用记录

ELK 平台收集Tomcat日志记录

elk日志分析搭建使用记录

elk日志分析搭建使用记录