flume 测试 hive sink

Posted 2022-04-28 hanwen1014

测试flume，将数据送到hive表中，首先建表。

create table order_flume(
order_id string,
user_id string,
eval_set string,
order_number string,
order_dow string,
order_hour_of_day string,
days_since_prior_order string)
clustered by (order_id) into 5 buckets
stored as orc; # 记得加 orc，不然后面会出错

　　flume conf 配置如下：

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -f /root/code/flume_exec_test.txt

# Describe the sink
a1.sinks.k1.type = hive
a1.sinks.k1.hive.metastore = thrift://master:9083
a1.sinks.k1.hive.database = hw
a1.sinks.k1.hive.table = order_flume
a1.sinks.k1.serializer = DELIMITED
a1.sinks.k1.serializer.delimiter = "," 
a1.sinks.k1.serializer.serdeSeparator = ‘,‘
a1.sinks.k1.serializer.fieldnames = order_id,user_id,eval_set,order_number,order_dow,order_hour_of_day,days_since_prior_order

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000 # 设置大一点，默认是1000
a1.channels.c1.transactionCapacity = 1000 # 设置大一点，默认是100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

　　这个时候如果启动flume的话会报错，需要将hive中的jar包移动到flume 中。

/usr/local/src/apache-hive-1.2.2-bin/hcatalog/share/hcatalog/* $FLUME_HOME/lib

/usr/local/src/apache-hive-1.2.2-bin/lib/* $FLUME_HOME/lib

　　此时，在修改修改 hive-site.xml，将下面的值进行修改。

    　　<property>
		<name>hive.support.concurrency</name>
		<value>true</value>
	</property>
	<property>
		<name>hive.exec.dynamic.partition.mode</name>
		<value>nonstrict</value>
	</property>
	<property>
		<name>hive.txn.manager</name>
		<value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>
	</property>
	<property>
		<name>hive.compactor.initiator.on</name>
		<value>true</value>
	</property>
	<property>
		<name>hive.compactor.worker.threads</name>
		<value>1</value>
	</property>

　　上面的配置完成之后，先启动 hive metastore，在启动 flume。

# 一定要重启
# 先重启 mysql
# 再重启 hadoop
# 再启动 hive metstore
# 再启动 flume

service mysql restart

start-dfs.sh
start-yarn.sh

hive --service metastore

flume-ng agent -c conf -f conf/hive.conf -n a1 -Dflume.root.logger=INFO,console

　　一开始调试的时候，配置的都是对的，无论怎么跑都是连不上，都提示失败。

org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint metaStoreUri=‘thrift://master:9083‘, database=‘hw‘, table=‘order_flume‘, partitionVals=[] 
	at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
	at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
	at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
	at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flume.sink.hive.HiveWriter$TxnBatchException: Failed acquiring Transaction Batch from EndPoint: metaStoreUri=‘thrift://master:9083‘, database=‘hw‘, table=‘oreder_flume‘, partitionVals=[] 
	at org.apache.flume.sink.hive.HiveWriter.nextTxnBatch(HiveWriter.java:400)
	at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:90)
	... 6 more
Caused by: org.apache.hive.hcatalog.streaming.TransactionBatchUnAvailable: Unable to acquire transaction batch on end point: metaStoreUri=‘thrift://master:9083‘, database=‘hw‘, table=‘order_flume‘, partitionVals=[] 
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:514)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:464)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatchImpl(HiveEndPoint.java:351)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatch(HiveEndPoint.java:331)
	at org.apache.flume.sink.hive.HiveWriter$9.call(HiveWriter.java:395)
	at org.apache.flume.sink.hive.HiveWriter$9.call(HiveWriter.java:392)
	at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused by: org.apache.thrift.TApplicationException: Internal error processing open_txns
	at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_open_txns(ThriftHiveMetastore.java:4195)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.open_txns(ThriftHiveMetastore.java:4182)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.openTxns(HiveMetaStoreClient.java:1988)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.openTxnImpl(HiveEndPoint.java:523)
	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:507)
	... 10 more

　　重启了几次还是同样的错误，后面尝试先启动 hive，去看一下其中表的情况，发现启动hive失败。

FAILED: LockException [Error 10280]: Error communicating with the metastore

　　再次尝试重启 metastore，在尝试启动 hive，一开始尝试2次，发现还是同样的错误，结果过了几分钟，再次重新启动后，发现它自己好了，一直没有弄明白为什么，可能是之前没重启成功，重新编辑一下 hive-site.xml文件，在次保存退出，在重启可能会好。

最终flume 链接 hive 成功，信息如下。

2019-07-20 12:26:28,968 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:342)] k1: Creating Writer to Hive end point : metaStoreUri=‘thrift://master:9083‘, database=‘hw‘, table=‘order_flume‘, partitionVals=[] 
2019-07-20 12:26:30,747 (hive-k1-call-runner-0) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:377)] Trying to connect to metastore with URI thrift://master:9083
2019-07-20 12:26:30,865 (hive-k1-call-runner-0) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:473)] Connected to metastore.
2019-07-20 12:26:31,494 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:377)] Trying to connect to metastore with URI thrift://master:9083
2019-07-20 12:26:31,516 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:473)] Connected to metastore.
2019-07-20 12:26:35,677 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hive.HiveWriter.nextTxnBatch(HiveWriter.java:335)] Acquired Txn Batch TxnIds=[1...100] on endPoint = metaStoreUri=‘thrift://master:9083‘, database=‘hw‘, table=‘order_flume‘, partitionVals=[] . Switching to first txn
2019-07-20 12:26:38,808 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hive.HiveWriter.commitTxn(HiveWriter.java:275)] Committing Txn 1 on EndPoint: metaStoreUri=‘thrift://master:9083‘, database=‘hw‘, table=‘order_flume‘, partitionVals=[]