使用 Ibis python 将数据插入到 impala 表中

Posted 2023-03-22

【中文标题】使用 Ibis python 将数据插入到 impala 表中

【英文标题】：Inserting data to impala table using Ibis python

【发布时间】：2017-11-22 22:54:06

【问题描述】：

我正在尝试将 df 插入到 ibis 创建的带有分区的 impala 表中。我在远程内核上运行它，在 Windows 10 机器上使用 spyder 3.2.4，在运行 CentOS 的边缘节点机器上使用 python 3.6.2。

我收到以下错误：

Writing DataFrame to temporary file
Writing CSV to: /tmp/ibis/pandas_0032f9dd1916426da62c8b4d8f4dfb92/0.csv
Traceback (most recent call last):

File "/usr/local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)

File "", line 1, in 
insert = target_table.insert(df3)

File "/usr/local/lib/python3.6/site-packages/ibis/impala/client.py", line 1674, in insert
writer, expr = write_temp_dataframe(self._client, obj)

File "/usr/local/lib/python3.6/site-packages/ibis/impala/pandas_interop.py", line 225, in write_temp_dataframe
return writer, writer.delimited_table(path)

File "/usr/local/lib/python3.6/site-packages/ibis/impala/pandas_interop.py", line 188, in delimited_table
schema = self.get_schema()

File "/usr/local/lib/python3.6/site-packages/ibis/impala/pandas_interop.py", line 184, in get_schema
return pandas_to_ibis_schema(self.df)

File "/usr/local/lib/python3.6/site-packages/ibis/impala/pandas_interop.py", line 219, in pandas_to_ibis_schema
return schema(pairs)

File "/usr/local/lib/python3.6/site-packages/ibis/expr/api.py", line 105, in schema
return Schema.from_tuples(pairs)

File "/usr/local/lib/python3.6/site-packages/ibis/expr/datatypes.py", line 109, in from_tuples
return Schema(names, types)

File "/usr/local/lib/python3.6/site-packages/ibis/expr/datatypes.py", line 55, in init
self.types = [validate_type(typ) for typ in types]

File "/usr/local/lib/python3.6/site-packages/ibis/expr/datatypes.py", line 55, in 
self.types = [validate_type(typ) for typ in types]

File "/usr/local/lib/python3.6/site-packages/ibis/expr/datatypes.py", line 1040, in validate_type
return TypeParser(t).parse()

File "/usr/local/lib/python3.6/site-packages/ibis/expr/datatypes.py", line 901, in parse
t = self.type()

File "/usr/local/lib/python3.6/site-packages/ibis/expr/datatypes.py", line 1033, in type
raise SyntaxError('Type cannot be parsed: '.format(self.text))

File "", line unknown
SyntaxError: Type cannot be parsed: integer

【参考方案1】：

由于 hadoop 系统的结构和安全性而出现错误。 Ibis 包尝试分别在 __ibis_tmp 和 /tmp/ibis/ 中创建 temp_db 和 temp_hdfs_location。由于在我们的系统中，默认位置不对除 root/系统管理员以外的任何用户开放...插入命令在从 /tmp/ibis/ 获取数据到实际 db 时出错（仍然不清楚，但可能是通过 __ibis_tmp dbase）。一旦我们将 ibis 包的 config_init.py 文件编辑到允许的临时位置/db。它就像一个魅力。

【讨论】：

或者，使用以下内容：ibis.config_init.impala_temp_hdfs_path_doc =

【参考方案2】：

而不是编辑提到的 config_init.py

https://***.com/a/47543691/5485370

使用 ibis.options 分配临时数据库和路径更容易：

ibis.options.impala.temp_db = 'your_temp_db'
ibis.options.impala.temp_hdfs_path = 'your_temp_hdfs_path'