将 Hbase 表加载到 Spark -

Posted 2023-04-18

【中文标题】将 Hbase 表加载到 Spark -

【英文标题】：Loading Hbase table onto Spark -

【发布时间】：2015-01-06 06:31:31

【问题描述】：

我正在尝试来自 https://www.mapr.com/developercentral/code/loading-hbase-tables-spark#.VKtxqivF_fS 的示例。当我通过 HBase shell 检查时，该表已创建并插入行。但是创建 RDD 然后计数的下一步给出了以下错误。 感谢您的帮助。

java.lang.IllegalStateException: unread block data
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode
(ObjectInputStream.java:2421)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
org.apache.spark.serializer.JavaDeserializationStream.readObject
(JavaSerializer.scala:62)
org.apache.spark.serializer.JavaSerializerInstance.deserialize
(JavaSerializer.scala:87)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:159)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)

【参考方案1】：

问题通过使用 --jars 选项将 HBase jar 文件传递​​给工作人员得到解决。我之前只使用过 --driver-class-path。

如下图

spark-submit --master spark://sparkhost:7077\
 --class SimpleApp \
 --jars /home/hadoop/Spark/hbase-0.98.7-hadoop2/lib/hbase-client-0.98.7-hadoop2.jar,\
/home/hadoop/Spark/hbase-0.98.7-hadoop2/lib/hbase-server-0.98.7-hadoop2.jar,\
/home/hadoop/Spark/hbase-0.98.7-hadoop2/lib/hbase-protocol-0.98.7-hadoop2.jar,\
/home/hadoop/Spark/hbase-0.98.7-hadoop2/lib/hbase-common-0.98.7-hadoop2.jar,\
/home/hadoop/BigDataEDW/htrace-core-2.04.jar\
 /home/hadoop/BigDataEDW/hbase-spark_2.10-1.0.0-SNAPSHOT.jar