nginx access 日志 ELK 加入REDIS 防止日志丢失 filebeat7.4.2 发送日志到redis 6.2.2

Posted 2022-04-08 chenglei1208

1.nginx 日志格式转换为json

1.vim nginx.conf
新增如下第一种方式

log_format log_json  "@timestamp": "$time_local", 
"remote_addr": "$remote_addr", 
"referer": "$http_referer", 
"request": "$request", 
"status": $status, 
"bytes": $body_bytes_sent, 
"agent": "$http_user_agent", 
"x_forwarded": "$http_x_forwarded_for", 
"up_addr": "$upstream_addr",
"up_host": "$upstream_http_host",
"up_resp_time": "$upstream_response_time",
"request_time": "$request_time"
 ;


access_log  /usr/local/nginx/logs/access.log log_json;
#上面日志地址写你自己的安装地址

第二种
log_format main   "@timestamp":"$time_iso8601",
                    "@source":"$server_addr",
                    "hostname":"$hostname",
                    "remote_user":"$remote_user",
                    "ip":"$http_x_forwarded_for",
                    "client":"$remote_addr",
                    "request_method":"$request_method",
                    "scheme":"$scheme",
                    "domain":"$server_name",
                    "referer":"$http_referer",
                    "request":"$request_uri",
                    "requesturl":"$request",
                    "args":"$args",
                    "size":$body_bytes_sent,
                    "status": $status,
                    "responsetime":$request_time,
                    "upstreamtime":"$upstream_response_time",
                    "upstreamaddr":"$upstream_addr",
                    "http_user_agent":"$http_user_agent",
                    "http_cookie":"$http_cookie",
                    "https":"$https"
                    ;
access_log  /usr/local/nginx/logs/access.log  main;
#上面日志地址写你自己的安装地址

2.filebeat设置

1.进入安装目录
cd /usr/local/filebeat/filebeat-7.4.2-linux-x86_64/
2.配置文件
vim filebeat.yml
如下
#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/nginx/logs/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]

- type: log
  enabled: true
  paths:
     - /usr/local/nginx/logs/error.log
  tags: ["error"]
# Each - is an input. Most options can be set at the input level, so
#----------------------------- Logstash output --------------------------------
#output.logstash:
output.redis:
  hosts: ["10.10.201.184:6379"]
  password: "123456"
  key: "nginx-151"
  db: 0
  timeout: 5

 # The Logstash hosts

    启动filebeat

启动filebeat
1.进入你的安装目录
cd /usr/local/filebeat/filebeat-7.4.2-linux-x86_64
2.启动服务 （前台启动）
[root@localhost filebeat-7.4.2-linux-x86_64]# ./filebeat -e -c filebeat.yml

filbeat设置完成!

redis安装

一、安装依赖
1.gcc
yum -y install gcc
2.gcc++
yum -y install gcc-c++

二上传文件和解压缩 安装
3.上传软件 至/home/xiazai文件夹
redis-6.2.6.tar.gz
4.解压缩
cd /home/xiazai
tar -zxvf redis-6.2.6.tar.gz
5.进入解压缩目录
cd redis-6.2.6
6.
make
显示下面成功

Hint: Its a good idea to run make test ;)

make[1]: Leaving directory `/root/redis-3.2.10/redis-6.2.6/src

7.安装到/usr/local/redis
make install PREFIX=/usr/local/redis

8.从 redis 的源码目录中复制 redis.conf 到 redis 的安装目录
[root@localhost redis-6.2.6]# cp redis.conf /usr/local/redis/bin/

从 redis 源码的src目录中redis-cli redis-server 复制到/usr/bin/目录
[root@localhost redis-6.2.6]#  cd src
cp redis-cli /usr/bin
cp redis-server /usr/bin/

9.编辑配置文件
vim /usr/local/redis/bin/redis.conf

设置更改如下  其他默认 注意：requirepass 123456 是设置的密码

 bind 0.0.0.0
 requirepass 123456
protected-mode no
daemonize yes

10.启动服务
[root@localhost bin]# ./redis-server redis.conf
11.查看进程 已启动
[root@localhost bin]# ps -ef | grep redis
root      16349      1  0 15:20 ?        00:00:00 ./redis-server 0.0.0.0:6379
root      16355  11712  0 15:20 pts/0    00:00:00 grep --color=auto redis

11.
设置redis 开机启动
建立配置文件
 root@localhost bin] vim  /etc/systemd/system/redis.service　
如下

[Unit]
Description=redis-server
After=network.target
 
[Service]
Type=forking
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/bin/redis.conf
PrivateTmp=true
 
[Install]
WantedBy=multi-user.target

保存退出

12.redis设置开机启动
刷新一下
root@localhost bin]# systemctl daemon-reload
启动服务
[root@localhost bin]# systemctl start redis.service
设置开机启动
[root@localhost bin]# systemctl enable redis.service
显示如下 完成

Created symlink from /etc/systemd/system/multi-user.target.wants/redis.service to /etc/systemd/system/redis.service.

13.创建 redis 命令软链接
[root@localhost bin]# ln -s /usr/local/redis/bin/redis-cli /usr/bin/redis

安装完成

测试REDIS本身

测试
1.查看进程 有进程的话 杀掉
[root@localhost bin]# ps -ef | grep redis
root      16349      1  0 15:20 ?        00:00:00 ./redis-server 0.0.0.0:6379
root      16447  11712  0 15:30 pts/0    00:00:00 grep --color=auto redis

2.有进程的话先杀掉进程
[root@localhost bin]# kill -9 16349

3.再查看进程 没了
[root@localhost bin]# ps -ef | grep redis
root      16449  11712  0 15:30 pts/0    00:00:00 grep --color=auto redis

4.启动 redis 服务
[root@localhost bin]# systemctl start redis

5.查看redis 服务状态
[root@localhost bin]# systemctl status redis

6.测试连接
注意：123456是设置配置文件时候设置的密码
[root@localhost bin]# redis 
127.0.0.1:6379>  auth "123456"  
OK
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> set abc 123
OK
127.0.0.1:6379>

测试nginx的日志 filebeat 是不是已经发到redis

第一种方式测试
[root@localhost ~]# redis
127.0.0.1:6379>  auth "123456" 
OK
127.0.0.1:6379> scan 0
1) "0"
2) 1) "nginx-151"
127.0.0.1:6379>

第二种方式测试
[root@localhost src]# redis-cli -h 10.10.201.184  -a 123456
Warning: Using a password with -a or -u option on the command line interface may not be safe.
10.10.201.184:6379> keys *
1) "nginx-151"
10.10.201.184:6379>