Spark SQL CTE在查询中忽略命名空间
Posted
技术标签:
【中文标题】Spark SQL CTE在查询中忽略命名空间【英文标题】：Spark SQL CTE ignoring namespace in query 【发布时间】：2020-07-15 23:48:28 【问题描述】：在本地 spark-sql 或通过 pyspark spark.sql(...) 运行 Spark，如果我在查询中使用 CTE，然后使用不正确的命名空间/数据库引用 CTE，则查询工作正常（意外）。当我在生产环境中（在 Databricks 上）运行查询时，我得到一个 Table or view not found 错误（预期）。
意外通过行为可以通过spark-sql在本地重现：
WITH myview AS (
SELECT 1 AS column
)
SELECT
*
FROM
invalid_namespace.myview;
当我预计它失败时返回“1”。
有人可以帮我在本地解决这个问题，以便我们在部署之前进行适当的测试吗？
从终端重现的具体步骤：
$ spark-sql
...
spark-sql> WITH some_new_cte AS (SELECT 1 AS column)
> SELECT * FROM namespace_does_not_exist.some_new_cte;
...
1
Time taken: 2.294 seconds, Fetched 1 row(s)
spark-sql>
【问题讨论】：
您确定视图不存在于无效的命名空间中吗？ 100% 确定。我可以在完全空的 spark 设置上运行上述查询。 非常有趣。我希望表级别的名称空间解析是能够正常工作的事情之一。构建 SQL 解析器似乎非常基础。 【参考方案1】：如果看查询计划，其实解析失败
== Parsed Logical Plan ==
CTE [myview]
: +- SubqueryAlias `myview`
: +- Project [1 AS column#0]
: +- OneRowRelation
+- 'Project [*]
+- 'UnresolvedRelation `invalid_namespace`.`myview`
== Analyzed Logical Plan ==
column: int
Project [column#0]
+- SubqueryAlias `myview`
+- Project [1 AS column#0]
+- OneRowRelation
== Optimized Logical Plan ==
Project [1 AS column#0]
+- OneRowRelation
== Physical Plan ==
*(1) Project [1 AS column#0]
+- Scan OneRowRelation[]
您的查询返回“1”的原因是因为 spark 看到您的视图在同一个查询中，所以它只是忽略了您的命名空间。如果命名空间真的不存在，就会失败。
【讨论】：
是的，这也是我观察到的行为。如果您希望它失败，则必须将这两个语句分开。首先，通过create 语句注册临时视图，然后在下一条语句中调用select，这样它肯定会失败。
这是特定于版本的，所以我怀疑这是旧版本中的错误【参考方案2】：
这似乎是一个影响 Spark 2.4.0 到 2.4.5 版本的错误（我没有检查
2.4.0 意外成功：
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch
Compiled by user on 2018-10-29T06:22:05Z
Revision
Url
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 10:59:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1595008748636
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
1
Time taken: 1.822 seconds, Fetched 1 row(s)
spark-sql>
已测试但未显示：2.4.1、2.4.2、2.4.3、2.4.4（均意外成功）
2.4.5 意外成功：
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.5
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch HEAD
Compiled by user centos on 2020-02-02T19:38:06Z
Revision cee4ecbb16917fa85f02c635925e2687400aa56b
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 10:59:55 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1595008798737
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
1
Time taken: 2.155 seconds, Fetched 1 row(s)
spark-sql>
2.4.6 中的预期失败：
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.6
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch HEAD
Compiled by user holden on 2020-05-29T23:47:51Z
Revision 807e0a484d1de767d1f02bd8a622da6450bdf940
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 11:00:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark master: local[*], Application Id: local-1595008843321
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
20/07/17 11:00:44 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
20/07/17 11:00:44 WARN ObjectStore: Failed to get database bad_namespace, returning NoSuchObjectException
Error in query: Table or view not found: `bad_namespace`.`mycte`; line 1 pos 49;
'Project [*]
+- 'UnresolvedRelation `bad_namespace`.`mycte`
spark-sql>
3.0.0 中的预期失败：
$ spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.0.0
/_/
Using Scala version 2.12.10, OpenJDK 64-Bit Server VM, 1.8.0_262
Branch HEAD
Compiled by user ubuntu on 2020-06-06T11:32:25Z
Revision 3fdfce3120f307147244e5eaf46d61419a723d50
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
$ echo "WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;" | SPARK_CONF_DIR=spark_conf spark-sql
20/07/17 11:01:40 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/07/17 11:01:42 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
20/07/17 11:01:42 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
20/07/17 11:01:44 ERROR ObjectStore: Version information found in metastore differs 1.2.0 from expected schema version 2.3.0. Schema verififcation is disabled hive.metastore.schema.verification
20/07/17 11:01:44 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore georgeleslie-waksman@10.0.1.178
Spark master: local[*], Application Id: local-1595008901594
spark-sql> WITH mycte AS (SELECT 1 AS column) SELECT * FROM bad_namespace.mycte;
20/07/17 11:01:45 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
20/07/17 11:01:45 WARN ObjectStore: Failed to get database bad_namespace, returning NoSuchObjectException
Error in query: Table or view not found: bad_namespace.mycte; line 1 pos 49;
'Project [*]
+- 'UnresolvedRelation [bad_namespace, mycte]
spark-sql>
【讨论】：
以上是关于Spark SQL CTE在查询中忽略命名空间的主要内容，如果未能解决你的问题，请参考以下文章
sql server使用cte递归查询获取树形的父节点/子节点