使用kafka作为生产者生产数据到hdfs(单节点)
Posted zdy0521
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了使用kafka作为生产者生产数据到hdfs(单节点)相关的知识，希望对你有一定的参考价值。
查看kafka官网的userguide
agent.sources = kafkaSource
agent.channels = memoryChannel
agent.sinks = hdfsSink
agent.sources.kafkaSource.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.kafksSource.zookeeperConnect = 192.168.57.11:2181
agent.sources.kafkaSource.topic = tets_zhang
agent.sources.kafkaSource.groupld = test-consumer-group
agent.sources.kafkaSource.kafka.consumer.timeout.ms = 100
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100
agent.channels.memoryChannel.transactionCapacity = 100
agent.sinks.hdfsSink.type = hdfs
agent.sinks..hdfsSink.hdfs.path = hdfs://beicai/test/zhang
agent.sinks.hdfsSink.hdfs.writeFormat = Text
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs..rollSize = 1024
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.rollinterval = 60
agent.sinks.hdfsSink.hdfs.filePrefix = test
agent.sinks.hdfsSink.hdfs.fileSuffix = .data
agent.sinks.hdfsSink.hdfs.inUserPrefix = _
agent.sinks.hdfsSink.hdfs.inUserSuffix=
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.writeFormat = TEXT
agent.sinks.hdfsSink.hdfs.rollnterval = 1
agent.sinks.sink1.hdfs.filePrefix = A
agent.sources.kafkaSource.channels = memoryChannel
agent.sinks.hdfsSink.channel = memoryChanne
以上是关于使用kafka作为生产者生产数据到hdfs(单节点)的主要内容，如果未能解决你的问题，请参考以下文章