flume实战

Posted 2020-11-15 yeqian100

flume 三大组件

source 收集

channel 聚集

sink 输出

 

使用Flume关键就是写配置文件

A 配置source

B 配置channel

C 配置sink

D 把以上3个组件串起来

 

1.通过IP端口 接收数据

  

a1 agent名称
r1 数据源名称
k1 sinks名称
c1 channel名称

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind =hadoop000
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

 

启动agent

flume-ng agent 

--name a1

--conf  $FLUME_HOME/conf

--conf-file  $FLUME_HOME/conf/example.conf

-Dflume.roog.logger=INFO,console

 

agent选型 ： exec source+ memory channel + logger skin

# Name the components on this agent

a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command=tail -f /home/hadoop/data/data.log
a1.sources.r1.shell=/bin/sh -c

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1