pyspark.sql 无法实例化 HiveMetaStoreClient - noclassfound from org.apache.commons.dbcp.connectionfactory

Posted 2023-04-15

技术标签:

【中文标题】pyspark.sql 无法实例化 HiveMetaStoreClient - noclassfound from org.apache.commons.dbcp.connectionfactory【英文标题】：pyspark.sql Unable to instantiate HiveMetaStoreClient - noclassfound from org.apache.commons.dbcp.connectionfactory 【发布时间】：2020-04-21 16:26:49 【问题描述】：

使用 Pyspark (python 3.7.1) 连接到外部 hive postgres 元存储。运行 spark.sql("show databases") 时出现错误Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient - Caused by: noclassfound from org.apache.commons.dbcp.connectionfactory

连接到外部 hive postgres metastore 的 spark 配置参数如下面的代码 sn-p 所示。我添加了 postgres 库，在 config.xml 中设置了所有连接详细信息。配置中提到的用户名/密码是正确的。不过，还不清楚错误。详细的错误日志在下面的代码 sn-p 中。任何人都知道如何使用 spark.sql 进行查询？谢谢，

pyspark 代码 sn-p

from pyspark.sql import SparkSession
def spark_init():
    spark = (
    SparkSession.builder
    .config("spark.debug.maxToStringFields", "10000")
    .config("spark.hadoop.hive.exec.dynamic.partition.mode", "non-strict")
    .config("spark.hadoop.hive.exec.dynamic.partition", "true")
    .config("spark.sql.warehouse.dir","hdfs://xxx:8020/user/hive/warehouse")
    .config("hive.metastore.warehouse.dir", "hdfs://xxx:8020/user/hive/warehouse")
    .config("hive.exec.dynamic.partition.mode", "nonstrict")
    .config("hive.exec.dynamic.partition", "true")
    .config("spark.jars","/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/jars/postgresql-42.2.12.jar")
    .config("spark.hadoop.javax.jdo.option.ConnectionURL","jdbc:postgresql://xxx:5432/hive")
    .config("spark.hadoop.javax.jdo.option.ConnectionDriverName","org.postgresql.Driver")
    .config("spark.hadoop.javax.jdo.option.ConnectionUserName","hive")
    .config("spark.hadoop.javax.jdo.option.ConnectionPassword","hive")
    .enableHiveSupport()
    .getOrCreate()
    )

    return spark

spark = spark_init()

spark.sql("show databases").show();

错误日志

    Warning: Ignoring non-spark config property: hive.exec.dynamic.partition=true
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/04/21 12:55:34 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
<pyspark.sql.session.SparkSession object at 0x7fad44a927b8>
20/04/21 12:55:37 WARN metadata.Hive: Failed to register all functions.
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1775)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3819)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3871)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3851)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4105)
    at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254)
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237)
    at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:394)
    at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:338)
    at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:318)
    at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:294)
    at org.apache.spark.sql.hive.client.HiveClientImpl.org$apache$spark$sql$hive$client$HiveClientImpl$$client(HiveClientImpl.scala:254)
    at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:276)
    at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:221)
    at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:220)
    at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:266)
    at org.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:356)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:217)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:217)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:217)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:216)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.org$apache$spark$sql$hive$HiveSessionStateBuilder$$externalCatalog(HiveSessionStateBuilder.scala:39)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$1.apply(HiveSessionStateBuilder.scala:54)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$1.apply(HiveSessionStateBuilder.scala:54)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:90)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:90)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listDatabases(SessionCatalog.scala:242)
    at org.apache.spark.sql.execution.command.ShowDatabasesCommand$$anonfun$2.apply(databases.scala:44)
    at org.apache.spark.sql.execution.command.ShowDatabasesCommand$$anonfun$2.apply(databases.scala:44)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.command.ShowDatabasesCommand.run(databases.scala:44)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:651)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1773)
    ... 60 more
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/dbcp/ConnectionFactory
    at org.apache.hadoop.hive.metastore.datasource.DataSourceProviderFactory.<clinit>(DataSourceProviderFactory.java:30)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:607)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:655)
    at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:419)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:356)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:317)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:688)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:654)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:648)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:717)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:420)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7036)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
    ... 65 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.dbcp.ConnectionFactory
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:246)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:235)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 85 more
Traceback (most recent call last):
  File "/home/ergo.liferunoffinsuranceplatform.com/kdamarla/lip-core-dwh/src/python/venv37/lib/python3.7/site-packages/pyspark/sql/utils.py", line 63, in deco
    return f(*a, **kw)
  File "/home/ergo.liferunoffinsuranceplatform.com/kdamarla/lip-core-dwh/src/python/venv37/lib/python3.7/site-packages/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling o97.sql.
: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient;
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
    at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:216)
    at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:114)
    at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:102)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder.org$apache$spark$sql$hive$HiveSessionStateBuilder$$externalCatalog(HiveSessionStateBuilder.scala:39)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$1.apply(HiveSessionStateBuilder.scala:54)
    at org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$1.apply(HiveSessionStateBuilder.scala:54)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog$lzycompute(SessionCatalog.scala:90)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.externalCatalog(SessionCatalog.scala:90)
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.listDatabases(SessionCatalog.scala:242)
    at org.apache.spark.sql.execution.command.ShowDatabasesCommand$$anonfun$2.apply(databases.scala:44)
    at org.apache.spark.sql.execution.command.ShowDatabasesCommand$$anonfun$2.apply(databases.scala:44)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.execution.command.ShowDatabasesCommand.run(databases.scala:44)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
    at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$$anonfun$53.apply(Dataset.scala:3364)
    at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3363)
    at org.apache.spark.sql.Dataset.<init>(Dataset.scala:194)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:79)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:651)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:242)
    at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:394)
    at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:338)
    at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:318)
    at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:294)
    at org.apache.spark.sql.hive.client.HiveClientImpl.org$apache$spark$sql$hive$client$HiveClientImpl$$client(HiveClientImpl.scala:254)
    at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:276)
    at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:221)
    at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:220)
    at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:266)
    at org.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:356)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:217)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:217)
    at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:217)
    at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:99)
    ... 37 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1775)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:80)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:130)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:101)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3819)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3871)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3851)
    at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:4105)
    at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:254)
    at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:237)
    ... 51 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1773)
    ... 60 more
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/dbcp/ConnectionFactory
    at org.apache.hadoop.hive.metastore.datasource.DataSourceProviderFactory.<clinit>(DataSourceProviderFactory.java:30)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:607)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:655)
    at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:419)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:356)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:317)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:688)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:654)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:648)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:717)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:420)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:7036)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:254)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:70)
    ... 65 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.dbcp.ConnectionFactory
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:246)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:235)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 85 more'

【问题讨论】：

使用外部配置单元元存储，您可以通过配置单元命令提示符进行连接吗？我可以从 hive-shell 连接并显示数据库没有问题而不是通过 spark 传递 .config("spark.hadoop.javax.jdo.option.ConnectionPassword","hive") 您可以通过将其设置为类路径来使用 hive-site.xml 吗？是的，我也在 hive-site.xml 中传递了它们。也许会在配置中删除它们并让您知道结果。 @RamGhadiyaram 将它们从脚本中删除并仅保留在 hive-site.xml 中。仍然无法实例化配置单元元存储。引起：java.lang.ClassNotFoundException: org.apache.commons.dbcp.ConnectionFactory 【参考方案1】：

使用 Pyspark (python 3.7.1) 连接到 external hive postgres metastore。

您的 hive-site.xml 输入错误，或者 Metastore 服务已关闭。

可能是 hive 站点 xml 不在类路径中。您可以使用 spark spark.yarn.dist.files 并在此处指定 hive-site.xml 的路径。

将 hive-site.xml 从 $HIVE_HOME/conf/hive-site.xml 移动到 $SPARK_HOME/conf/hive-site.xml。在此文件中创建有关 hive Metastore uris 的条目。

# Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.
hive.metastore.uris thrift://<metastore-host>:<metastore-port>

示例 hive-site.xml

<configuration>

      <property>
        <name>hive.metastore.warehouse.dir</name>
        <value>/home/local/hive-metastore-dir/warehouse</value>

      </property>
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://192.168.122.1:9083</value>
</property>
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://localhost:3306/hivedb?createDatabaseIfNotExist=true</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>user</value>
    </property>
    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>password</value>
    </property>

    <configuration>

进一步检查wiki page

更新

找不到类是纯类路径问题，像这样使用

pyspark --jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/jars/postgresql-42.2.12.jar
//or
spark-submit --jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/jars/postgresql-42.2.12.jar <your py script or app jar>

如果你想添加所有的罐子而不遗漏任何东西，那么在下面使用


--jars $(echo //opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/jars/*.jar | tr ' ' ',')

这应该在你的类路径中也有你的 Commons dbcp jar。

如果jar文件会进入另一个文件夹，请不要从代码中发送所有参数，这是错误的方式，那么你需要更改代码。

【讨论】：

嗨@Ram，我有指定外部元存储详细信息的hive-site.xml。我重新启动了机器，我的错误进展到无法实例化 hive 元存储，其中未找到来自 org.apache.commons.dbcp.connectionfactory 的类异常。我也在用这个日志编辑实际问题。谢谢 thrift uri 中的 metastore-host 是外部 postgres ip-address 还是安装 hadoop 的 ip-address ？我的 hive-site.xml 中没有任何元存储 uri。用 postgres jar 提交了 spark-submit。仍然引起：java.lang.ClassNotFoundException：org.apache.commons.dbcp.ConnectionFactory 也许，我需要包含除 postgres 之外的任何其他 jar 吗？请不要通过更新不同的东西来逐步改变问题。我的答案将变得陈旧。因为你改变了上下文。为每个问题提出单独的问题当然@Ram。我会改变的。注册，包括所有罐子的最新更新，我可以提交所有罐子，如 spark-submit --jars=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/ jars/*.jar script.py ?

以上是关于pyspark.sql 无法实例化 HiveMetaStoreClient - noclassfound from org.apache.commons.dbcp.connectionfactory的主要内容，如果未能解决你的问题，请参考以下文章

是否可以在 Pyspark 中对 DataFrame 进行子类化？

nTypeError：无法合并类型 <class \'pyspark.sql.types.DoubleType\'> 和 <class \'pyspark.sql.types.Str

在pyspark中以分布式方式有效地生成大型DataFrame（没有pyspark.sql.Row）

在 PySpark SQL 中并行执行读写 API 调用

pyspark.sql.utils.AnalysisException：u'无法推断Parquet的模式。必须手动指定。

将pyspark列连接到pyspark DataFrame