如何从本地安装的 spark 连接到 aws-redshift？

Posted 2023-03-31

【中文标题】如何从本地安装的 spark 连接到 aws-redshift？

【英文标题】：how to connect from locally installed spark to aws-redshift?

【发布时间】：2021-05-28 15:26:48

【问题描述】：

下载了必要的库以从本地安装的 spark 集群连接 redshift，并使用以下命令启动 pyspark，但我收到以下错误消息。

 pyspark --conf spark.executor.extraClassPath=/usr/share/java/redshift-jdbc42-2.0.0.4.jar --driver-class-path /usr/share/java/redshift-jdbc42-2.0.0.4.jar  --jars /usr/share/java/redshift-jdbc42-2.0.0.4.jar



df = spark.read.format("jdbc").option("url", "jdbc:redshift://host/dbname").option("driver", "com.amazon.redshift.jdbc42.driver").option("dbtable", "tablename").option("user", "username").option("password", "password").load()

error 

 File "/usr/local/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o42.load.
: java.lang.ClassNotFoundException: com.amazon.redshift.jdbc42.driver
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.spark.sql.execution.datasources.jdbc.DriverRegistry$.register(DriverRegistry.scala:46)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1(JDBCOptions.scala:102)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$1$adapted(JDBCOptions.scala:102)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:102)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:38)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:354)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
    at org.apache.spark

【问题讨论】：

你有没有试过放置jars spark_home 路径，安装spark的地方，如果是linux或mac OS，你可以在/usr/local/Cellar/apache-spark/3.0.0/libexec/jars. 下找到路径并在那里添加jar，它应该可以工作。

【参考方案1】：

不是"com.amazon.redshift.jdbc42.driver"， 但是"com.amazon.redshift.jdbc42.Driver"。