08生产预警平台项目之Flume Agent(聚合节点) sink to kafka cluster

Posted 若泽大数据

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了08生产预警平台项目之Flume Agent(聚合节点) sink to kafka cluster相关的知识,希望对你有一定的参考价值。

1.创建logtopic
[root@sht-sgmhadoopdn-01 kafka]# bin/kafka-topics.sh --create --zookeeper 172.16.101.58:2181,172.16.101.59:2181,172.16.101.60:2181/kafka --replication-factor 3 --partitions 1 --topic logtopic


2.创建avro_memory_kafka.properties (kafka sink)
[root@sht-sgmhadoopcm-01 ~]# cd /tmp/flume-ng/conf
[root@sht-sgmhadoopcm-01 conf]# cp avro_memory_hdfs.properties avro_memory_kafka.properties
[root@sht-sgmhadoopcm-01 conf]# vi avro_memory_kafka.properties 
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 172.16.101.54
a1.sources.r1.port = 4545


# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.topic = logtopic
a1.sinks.k1.kafka.bootstrap.servers = 172.16.101.58:9092,172.16.101.59:9092,172.16.101.60:9092
a1.sinks.k1.kafka.flumeBatchSize = 6000
a1.sinks.k1.kafka.producer.acks = 1
a1.sinks.k1.kafka.producer.linger.ms = 1
a1.sinks.ki.kafka.producer.compression.type = snappy


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.keep-alive = 90
a1.channels.c1.capacity = 2000000
a1.channels.c1.transactionCapacity = 6000


# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1


3.后台启动 flume-ng agent(聚合节点)和查看nohup.out 
[root@sht-sgmhadoopcm-01 ~]# source /etc/profile
[root@sht-sgmhadoopcm-01 ~]# cd /tmp/flume-ng/
[root@sht-sgmhadoopcm-01 flume-ng]# nohup  flume-ng agent -c conf -f /tmp/flume-ng/conf/avro_memory_kafka.properties  -n a1 -Dflume.root.logger=INFO,console &
[1] 4971
[root@sht-sgmhadoopcm-01 flume-ng]# nohup: ignoring input and appending output to `nohup.out'

[root@sht-sgmhadoopcm-01 flume-ng]# 
[root@sht-sgmhadoopcm-01 flume-ng]# 
[root@sht-sgmhadoopcm-01 flume-ng]# cat nohup.out


4.检查log收集的三台(收集节点)开启没
[hdfs@flume-agent-01 flume-ng]$ . ~/.bash_profile 
[hdfs@flume-agent-02 flume-ng]$ . ~/.bash_profile 
[hdfs@flume-agent-03 flume-ng]$ . ~/.bash_profile


[hdfs@flume-agent-01 flume-ng]$ nohup  flume-ng agent -c /tmp/flume-ng/conf -f /tmp/flume-ng/conf/exec_memory_avro.properties -n a1 -Dflume.root.logger=INFO,console &
[hdfs@flume-agent-01 flume-ng]$ nohup  flume-ng agent -c /tmp/flume-ng/conf -f /tmp/flume-ng/conf/exec_memory_avro.properties -n a1 -Dflume.root.logger=INFO,console &
[hdfs@flume-agent-01 flume-ng]$ nohup  flume-ng agent -c /tmp/flume-ng/conf -f /tmp/flume-ng/conf/exec_memory_avro.properties -n a1 -Dflume.root.logger=INFO,console &


5.打开kafka manager监控
http://172.16.101.55:9999 


 如需转载,请在开篇显著位置注明文章出处(转自:若泽大数据 ruozedata

· 无原创标识文章请按照原文内容编辑,可直接转载,转载后请将转载链接发送给我们;

· 联系邮箱:2643854124@qq.com






以上是关于08生产预警平台项目之Flume Agent(聚合节点) sink to kafka cluster的主要内容,如果未能解决你的问题,请参考以下文章

21生产预警平台项目之记录一个flume-ng的tail -f参数所诱发的血案

02生产预警平台项目之Flume-1.7.0源码编译导入eclipse

24生产预警项目平台之Tomcat的支持log4j,日志输出为json格式

flume系列之:python读取flume配置文件,并把配置写入到zookeeper节点,再根据写入到zookeeper中的配置启动flume agent

大数据架构之:Flume

日志收集系统之Apache Flume