Centos 7.4下安装FileBeat

Posted 2023-01-01 wintersweet321

一. 直接yum方式安装

 1 [root@LogServer /]# yum install filebeat
 2 Loaded plugins: fastestmirror, langpacks
 3 base                                                                                                                                             | 3.6 kB  00:00:00     
 4 docker-ce-stable                                                                                                                                 | 3.5 kB  00:00:00     
 5 elastic-7.x                                                                                                                                      | 1.3 kB  00:00:00     
 6 extras                                                                                                                                           | 3.4 kB  00:00:00     
 7 rsyslog_v8                                                                                                                                       | 2.5 kB  00:00:00     
 8 updates                                                                                                                                          | 3.4 kB  00:00:00     
 9 Loading mirror speeds from cached hostfile
10  * base: ap.stykers.moe
11  * extras: centos.ustc.edu.cn
12  * updates: ap.stykers.moe
13 Resolving Dependencies
14 --> Running transaction check
15 ---> Package filebeat.x86_64 0:7.4.0-1 will be installed
16 --> Finished Dependency Resolution
17 
18 Dependencies Resolved
19 
20 ========================================================================================================================================================================
21  Package                                 Arch                                  Version                                 Repository                                  Size
22 ========================================================================================================================================================================
23 Installing:
24  filebeat                                x86_64                                7.4.0-1                                 elastic-7.x                                 23 M
25 
26 Transaction Summary
27 ========================================================================================================================================================================
28 Install  1 Package
29 
30 Total download size: 23 M
31 Installed size: 74 M
32 Is this ok [y/d/N]: y
33 Downloading packages:
34 filebeat-7.4.0-x86_64.rpm                                                                                                                        |  23 MB  00:00:07     
35 Running transaction check
36 Running transaction test
37 Transaction test succeeded
38 Running transaction
39   Installing : filebeat-7.4.0-1.x86_64                                                                                                                              1/1 
40   Verifying  : filebeat-7.4.0-1.x86_64                                                                                                                              1/1 
41 
42 Installed:
43   filebeat.x86_64 0:7.4.0-1                                                                                                                                             
44 
45 Complete!

View Code

 

二. 进入/etc/filebeat下，修改配置文件filebeat.yml，结果如下：

 1 [root@LogServer filebeat]# egrep -v "#|^$" /etc/filebeat/filebeat.yml
 2 filebeat.inputs:
 3 - type: log
 4   enabled: true
 5   paths:
 6     - /data/log/127.0.0.1/*.log    ##此处是rsyslog日志服务器集中采集本地日志文件
 7 filebeat.config.modules:
 8   path: $path.config/modules.d/*.yml
 9   reload.enabled: false
10 setup.template.settings:
11   index.number_of_shards: 1
12 setup.kibana:
13 output.elasticsearch:
14   hosts: ["localhost:9200"]

 

三. 官网给出的检查配置文件的TIPS

To test your configuration file, change to the directory where the Filebeat binary is installed, and run Filebeat in the foreground with the following options specified: ./filebeat test config -e. Make sure your config files are in the path expected by Filebeat (see Directory layout), or use the -c flag to specify the path to the config file.

 1 [root@LogServer bin]# ./filebeat test config -e -c /etc/filebeat/filebeat.yml
 2 2019-10-05T21:04:54.678+0800    INFO    instance/beat.go:607    Home path: [/usr/share/filebeat/bin] Config path: [/usr/share/filebeat/bin] Data path: [/usr/share/filebeat/bin/data] Logs path: [/usr/share/filebeat/bin/logs]
 3 2019-10-05T21:04:54.678+0800    INFO    instance/beat.go:615    Beat ID: 4f580ab1-f94b-44aa-99f4-364e6cf1d3b0
 4 2019-10-05T21:04:54.679+0800    INFO    [beat]  instance/beat.go:903    Beat info       "system_info": "beat": "path": "config": "/usr/share/filebeat/bin", "data": "/usr/share/filebeat/bin/data", "home": "/usr/share/filebeat/bin", "logs": "/usr/share/filebeat/bin/logs", "type": "filebeat", "uuid": "4f580ab1-f94b-44aa-99f4-364e6cf1d3b0"
 5 2019-10-05T21:04:54.679+0800    INFO    [beat]  instance/beat.go:912    Build info      "system_info": "build": "commit": "f940c36884d3749901a9c99bea5463a6030cdd9c", "libbeat": "7.4.0", "time": "2019-09-27T07:45:44.000Z", "version": "7.4.0"
 6 2019-10-05T21:04:54.679+0800    INFO    [beat]  instance/beat.go:915    Go runtime info "system_info": "go": "os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.9"
 7 2019-10-05T21:04:54.681+0800    INFO    [beat]  instance/beat.go:919    Host info       "system_info": "host": "architecture":"x86_64","boot_time":"2019-09-30T14:57:02+08:00","containerized":false,"name":"LogServer","ip":["127.0.0.1/8","::1/128","172.33.7.51/24","fe80::a65d:781a:3778:5b17/64","172.17.0.1/16"],"kernel_version":"3.10.0-693.el7.x86_64","mac":["28:6e:d4:88:c7:e1","28:6e:d4:88:c7:e2","02:42:79:dc:1e:df"],"os":"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":4,"patch":1708,"codename":"Core","timezone":"CST","timezone_offset_sec":28800,"id":"be94151089484f0d834d4e8a2e118751"
 8 2019-10-05T21:04:54.681+0800    INFO    [beat]  instance/beat.go:948    Process info    "system_info": "process": "capabilities": "inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null, "cwd": "/usr/share/filebeat/bin", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 105115, "ppid": 104989, "seccomp": "mode":"disabled", "start_time": "2019-10-05T21:04:53.660+0800"
 9 2019-10-05T21:04:54.681+0800    INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.4.0
10 2019-10-05T21:04:54.681+0800    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to ‘filebeat-7.4.0‘ as ILM is enabled.
11 2019-10-05T21:04:54.681+0800    INFO    elasticsearch/client.go:170     Elasticsearch url: http://localhost:9200
12 2019-10-05T21:04:54.682+0800    INFO    [publisher]     pipeline/module.go:97   Beat name: LogServer
13 2019-10-05T21:04:54.682+0800    ERROR   fileset/modules.go:125  Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
14 Config OK

View Code

 

四. 加入自启动

[root@LogServer /]# /bin/systemctl daemon-reload
[root@LogServer /]# /bin/systemctl enable filebeat.service
Created symlink from /etc/systemd/system/multi-user.target.wants/filebeat.service to /usr/lib/systemd/system/filebeat.service.

 

五. 启动服务

 1 [root@LogServer /]# systemctl start filebeat.service
 2 [root@LogServer /]# systemctl status filebeat.service
 3 ? filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
 4    Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
 5    Active: active (running) since Sat 2019-10-05 21:14:38 CST; 4s ago
 6      Docs: https://www.elastic.co/products/beats/filebeat
 7  Main PID: 105264 (filebeat)
 8    CGroup: /system.slice/filebeat.service
 9            +-105264 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/f...
10 
11 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.943-0400        INFO        instance/beat.go:422        filebeat start running.
12 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.943-0400        INFO        registrar/registrar.go:145        Loading registrar data from /.../data.json
13 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.943-0400        INFO        registrar/registrar.go:152        States Loaded from registrar: 8
14 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.944-0400        INFO        crawler/crawler.go:72        Loading Inputs: 1
15 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.944-0400        INFO        [monitoring]        log/log.go:118        Starting metrics logging every 30s
16 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.945-0400        INFO        log/input.go:152        Configured paths: [/var/log/*.log]
17 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.945-0400        INFO        input/input.go:114        Starting input of type: log; ID: 1120...9762598069
18 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.945-0400        INFO        crawler/crawler.go:106        Loading and starting Inputs compl... inputs: 1
19 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.945-0400        INFO        cfgfile/reload.go:171        Config reloader started
20 Oct 05 21:14:38 LogServer filebeat[105264]: 2019-10-05T09:14:38.946-0400        INFO        cfgfile/reload.go:226        Loading of config files completed.
21 Hint: Some lines were ellipsized, use -l to show in full.

View Code

 

六. 进程检查

 

1 [root@LogServer zhangxm]# ps -aux | grep filebeat
2 root     109342  0.0  0.6 1608036 52800 ?       Ssl  Oct06   1:17 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
3 root     117142  0.0  0.0 112664   980 pts/0    S+   11:44   0:00 grep --color=auto filebeat

 

七. 检查启动成功后，到es服务器上查看索引，可以看到新增了一个以filebeat-7.4.0开头的索引，这就代表filesbeat和es能够正常通信了

1 [root@LogServer zhangxm]# curl ‘localhost:9200/_cat/indices?v‘
2 health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
3 green  open   .kibana_task_manager_1           rIJCut09QuGYfuRbTUr2Wg   1   0          2            0     12.5kb         12.5kb
4 green  open   .apm-agent-configuration         TC9wWs0-R9Owna4G1kkuUw   1   0          0            0       283b           283b
5 green  open   .kibana_1                        9AfuMpbeS7mrVGti3rc0pw   1   0          2            0     11.2kb         11.2kb
6 yellow open   filebeat-7.4.0-2019.10.05-000001 2nhRfhN8RCG4DO6JaXeoaA   1   1     133314            0     33.9mb         33.9mb

 

八. 获取指定索引详细信息

 1 [root@LogServer zhangxm]# curl -XGET ‘localhost:9200/filebeat-7.4.0-2019.10.05-000001?pretty‘
 2 
 3   "filebeat-7.4.0-2019.10.05-000001" : 
 4     "aliases" : 
 5       "filebeat-7.4.0" : 
 6         "is_write_index" : true
 7       
 8     ,
 9     "mappings" : 
10       "_meta" : 
11         "beat" : "filebeat",
12         "version" : "7.4.0"
13       ,
14       "dynamic_templates" : [
15         
16           "labels" : 
17             "path_match" : "labels.*",
18             "match_mapping_type" : "string",
19             "mapping" : 
20               "type" : "keyword"
21             
22           
23         ......

 

 

 

Default paths

Filebeat uses the following default paths unless you explicitly change them.

 

Type	Description	Location
home	Home of the Filebeat installation.	/usr/share/filebeat
bin	The location for the binary files.	/usr/share/filebeat/bin
config	The location for configuration files.	/etc/filebeat
data	The location for persistent data files.	/var/lib/filebeat
logs	The location for the logs created by Filebeat.	/var/log/filebeat

For the deb and rpm distributions, these paths are set in the init script or in the systemd unit file. Make sure that you start the Filebeat service by using the preferred operating system method (init scripts or systemctl). Otherwise the paths might be set incorrectly.