无法使用pyspark运行简单的hql文件[重复]

Posted 2023-04-17

【中文标题】无法使用pyspark运行简单的hql文件[重复]

【英文标题】：Cannot run simple hql file with pyspark [duplicate]

【发布时间】：2020-01-25 23:36:58

【问题描述】：

我正在使用 pyspark==2.4.3，我只想运行一个 hql 文件

use myDatabaseName;
show tables;

这是我尝试过的

from os.path import expanduser, join, abspath

from pyspark.sql import SparkSession
from pyspark.sql import Row

# warehouse_location points to the default location for managed databases and tables
warehouse_location = abspath('spark-warehouse')

spark = SparkSession \
    .builder \
    .appName("Python Spark SQL Hive integration example") \
    .config("spark.sql.warehouse.dir", warehouse_location) \
    .enableHiveSupport() \
    .getOrCreate()

with open('full/path/to/my/hqlfile') as t:
    q=t.read()

print q
'use myDatabaseName;show tables;\n'
spark.sql(q)

但我明白了

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/some/path/python2.7/site-packages/pyspark/sql/session.py", line 767, in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
  File "/some/path/python2.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/some/path/python2.7/site-packages/pyspark/sql/utils.py", line 73, in deco
    raise ParseException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.ParseException: u"\nmismatched input ';' expecting <EOF>(line 1, pos 11)\n\n== SQL ==\nuse myDatabaseName;show tables;\n-----------^^^\n"

我做错了什么？

【参考方案1】：

与提示的错误一样，; 在 spark.sql 中的语法无效，

其次，你不能在一个 spark.sql 调用中调用两个命令。

我会将q 修改为不包含; 的查询字符串列表，然后进行for 循环。

query_lt = q.split(";")[:-1]
for qs in query_lt:
    spark.sql(qs)