CHD-5.3.6集群上Flume安装
Posted hello-wei
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了CHD-5.3.6集群上Flume安装相关的知识，希望对你有一定的参考价值。
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
翻译：
Flume是一种分布式、可靠且可用的服务，用于高效地收集、聚合和移动大量日志数据。它具有基于流数据流的简单灵活的体系结构。它具有鲁棒性和容错性，具有可调的可靠性机制和许多故障转移和恢复机制。它使用一个简单的可扩展数据模型，允许在线分析应用程序。
web Server 端产生日志，Source到具体目录下获取日志文件，把日志Channel中，Sink写到HDFS上
Source → Channel 可以进行数据清洗
Channel → Sink 可以进行数据清洗
vent是Flume数据传输的基本单元
Flume以事件的形式将数据从源头传送到最终的目的
Event由可选的header和载有数据的一个byte array构成
载有的数据对flume是不透明的
Header是容纳了key-value字符串对的无序集合，key在集合内是唯 一的。
Header可以在上下文路由中使用扩展
前提依赖：
* 运行在有log的地方
* 运行在LINUX
* JVM
解压：flume-ng-1.5.0-cdh5.3.6.tar.gz
mv apache-flume-1.5.0-cdh5.3.6-bin/ flume-1.5.0-cdh5.3.6
cd /home/hadoop/CDH5.3.6/flume-1.5.0-cdh5.3.6/conf
cp flume-env.sh.template flume-env.sh
vi flume-env.sh
export JAVA_HOME=/usr/local/jdk1.8
[[email protected] flume-1.5.0-cdh5.3.6]$ bin/flume-ng Usage: bin/flume-ng <command> [options]... commands: agent run a Flume agent avro-client run an avro Flume client version show Flume version info global options: --conf,-c <conf> use configs in <conf> directory-Dproperty=value sets a Java system property value agent options: --name,-n <name> the name of this agent (required) --conf-file,-f <file> specify a config file (required if -z missing)
运行命令：
bin/flume-ng agent --conf conf --name agent-test --conf-file test.conf
bin/flume-ng agent -c conf -n agent-test -f test.conf
配置一个a.conf
# The configuration file needs to define the sources, # the channels and the sinks. # Sources, channels and sinks are defined per agent, # in this case called ‘agent‘ ###define agent a1.sources = r1 a1.channels = c1 a1.sinks = k1 ### define sources a1.sources.r1.type = netcat a1.sources.r1.bind = master a1.sources.r1.port = 44444 ### define channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 ### define sink a1.sinks.k1.type=logger a1.sinks.k1.maxBytyesToLog = 2014 ### bind the source and sinks to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
安装 xinetd
[[email protected] telnet]# ll total 224 ----rwxr-x. 1 hadoop hadoop 59120 Jun 22 23:49 telnet-0.17-47.el6_3.1.x86_64.rpm ----rwxr-x. 1 hadoop hadoop 37748 Jun 22 23:49 telnet-server-0.17-47.el6_3.1.x86_64.rpm ----rwxr-x. 1 hadoop hadoop 124280 Jun 22 23:49 xinetd-2.3.14-38.el6.x86_64.rpm [[email protected] telnet]# rpm -ivh *.rpm warning: telnet-0.17-47.el6_3.1.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY Preparing... ########################################### [100%] 1:xinetd ########################################### [ 33%] 2:telnet-server ########################################### [ 67%] 3:telnet ########################################### [100%] [[email protected] telnet]#
启动服务：
[[email protected] telnet]# /etc/rc.d/init.d/xinetd restart Stopping xinetd: [FAILED] Starting xinetd: [ OK ] [[email protected] telnet]#
运行命令：
bin/flume-ng agent -c conf -n a1 -f conf/a.conf -D flume.root.logger=DEBUG,consol
查看端口：
[[email protected] flume-1.5.0-cdh5.3.6]# netstat -tnlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.1.30:19888 0.0.0.0:* LISTEN 3735/java tcp 0 0 0.0.0.0:10033 0.0.0.0:* LISTEN 3735/java tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 2715/java tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1924/sshd tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1685/cupsd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2299/master tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 2815/java tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 2815/java tcp 0 0 192.168.1.30:10020 0.0.0.0:* LISTEN 3735/java tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 2815/java tcp 0 0 192.168.1.30:9000 0.0.0.0:* LISTEN 2715/java tcp 0 0 192.168.1.30:50090 0.0.0.0:* LISTEN 2990/java tcp 0 0 :::22 :::* LISTEN 1924/sshd tcp 0 0 ::1:631 :::* LISTEN 1685/cupsd tcp 0 0 ::1:25 :::* LISTEN 2299/master tcp 0 0 ::ffff:192.168.1.30:44444 :::* LISTEN 17488/java tcp 0 0 :::3306 :::* LISTEN 2152/mysqld
运行命令：bin/flume-ng agent -c conf -n a1 -f conf/a.conf -Dflume.root.logger=DEBUG,console
在另一个控制台
[[email protected] ~]$ telnet master 44444 Trying 192.168.1.30... Connected to master. Escape character is ‘^]‘. hello flume OK hello world OK
在控制台的运行命令下，会出现：
以上是关于CHD-5.3.6集群上Flume安装的主要内容，如果未能解决你的问题，请参考以下文章