Flume总结

Posted 2020-09-17

一、日志采集：从网络端口接收数据，下沉到logger

文件netcat-logger.conf:

 1 # Name the components on this agent
 2 #给那三个组件取个名字
 3 a1.sources = r1
 4 a1.sinks = k1
 5 a1.channels = c1
 6 
 7 # Describe/configure the source
 8 #类型, 从网络端口接收数据,在本机启动, 所以localhost, type=spoolDir采集目录源,目录里有就采
 9 a1.sources.r1.type = netcat
10 a1.sources.r1.bind = localhost
11 a1.sources.r1.port = 44444
12 
13 # Describe the sink
14 a1.sinks.k1.type = logger
15 
16 # Use a channel which buffers events in memory
17 #下沉的时候是一批一批的, 下沉的时候是一个个eventChannel参数解释：
18 #capacity：默认该通道中最大的可以存储的event数量
19 #trasactionCapacity：每次最大可以从source中拿到或者送到sink中的event数量
20 a1.channels.c1.type = memory
21 a1.channels.c1.capacity = 1000
22 a1.channels.c1.transactionCapacity = 100
23 
24 # Bind the source and sink to the channel
25 a1.sources.r1.channels = c1
26 a1.sinks.k1.channel = c1

启动命令：
#告诉flum启动一个agent,指定配置参数, --name:agent的名字,
flume-ng agent --conf conf --conf-file conf/netcat-logger.conf --name a1 -Dflume.root.logger=INFO,console

传入数据：

[[email protected] ~]# telnet localhost 44444

Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is ‘^]‘.
hello world!^H^H^H^H^H^H^H^H^H^H^H^H^H^H
OK
tianjun2012!
OK

控制台看到的数据

2017-05-08 13:41:35,766 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 21 08 08 08 08 hello world!.... }
2017-05-08 13:41:40,153 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 74 69 61 6E 6A 75 6E 32 30 31 32 21 0D tianjun2012!. }