雪花 Python 连接 KeyError: 'snowflake-connector-python

Posted 2023-05-08

【中文标题】雪花 Python 连接 KeyError: \'snowflake-connector-python

【英文标题】：snowflake Python Connection KeyError: 'snowflake-connector-python雪花 Python 连接 KeyError: 'snowflake-connector-python

【发布时间】：2020-09-18 10:25:21

【问题描述】：

我已经安装了雪花 sqlachemy 包。并成功开发了将csv文件上传到雪花的脚本。

但是当我将脚本上传到aws glue并提供wheel文件时，我遇到了KeyError: 'snowflake-connector-python。

有什么想法吗？

这里是完整的错误信息：

Traceback (most recent call last):
  File "/tmp/runscript.py", line 123, in <module>
    runpy.run_path(temp_file_path, run_name='__main__')
  File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 250, in <module>
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 39, in main
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 233, in load_data
  File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 479, in create_engine
    return strategy.create(*args, **kwargs)
  File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 222, in load
    return self.impls[name]()
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 245, in load
    mod = compat.import_(modulepath)
  File "/glue/lib/installation/snowflake/sqlalchemy/__init__.py", line 8, in <module>
    from . import snowdialect
  File "/glue/lib/installation/snowflake/sqlalchemy/snowdialect.py", line 27, in <module>
    from snowflake.connector.constants import UTF8
  File "/glue/lib/installation/snowflake/connector/__init__.py", line 17, in <module>
    from .connection import SnowflakeConnection
  File "/glue/lib/installation/snowflake/connector/connection.py", line 43, in <module>
    from .cursor import LOG_MAX_QUERY_LENGTH, SnowflakeCursor
  File "/glue/lib/installation/snowflake/connector/cursor.py", line 47, in <module>
    from .arrow_result import ArrowResult
  File "src/snowflake/connector/arrow_result.pyx", line 16, in init snowflake.connector.arrow_result
  File "/glue/lib/installation/snowflake/connector/options.py", line 36, in <module>
    _pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']
KeyError: 'snowflake-connector-python'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/runscript.py", line 142, in <module>
    raise e_type(e_value).with_traceback(new_stack)
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 250, in <module>
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 39, in main
  File "/tmp/glue-python-scripts-e14jn3mj/etl_job_pchome_supply_chain.py", line 233, in load_data
  File "/glue/lib/installation/sqlalchemy/engine/__init__.py", line 479, in create_engine
    return strategy.create(*args, **kwargs)
  File "/glue/lib/installation/sqlalchemy/engine/strategies.py", line 61, in create
    entrypoint = u._get_entrypoint()
  File "/glue/lib/installation/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
    cls = registry.load(name)
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 222, in load
    return self.impls[name]()
  File "/glue/lib/installation/sqlalchemy/util/langhelpers.py", line 245, in load
    mod = compat.import_(modulepath)
  File "/glue/lib/installation/snowflake/sqlalchemy/__init__.py", line 8, in <module>
    from . import snowdialect
  File "/glue/lib/installation/snowflake/sqlalchemy/snowdialect.py", line 27, in <module>
    from snowflake.connector.constants import UTF8
  File "/glue/lib/installation/snowflake/connector/__init__.py", line 17, in <module>
    from .connection import SnowflakeConnection
  File "/glue/lib/installation/snowflake/connector/connection.py", line 43, in <module>
    from .cursor import LOG_MAX_QUERY_LENGTH, SnowflakeCursor
  File "/glue/lib/installation/snowflake/connector/cursor.py", line 47, in <module>
    from .arrow_result import ArrowResult
  File "src/snowflake/connector/arrow_result.pyx", line 16, in init snowflake.connector.arrow_result
  File "/glue/lib/installation/snowflake/connector/options.py", line 36, in <module>
    _pandas_extras = pkg_resources.working_set.by_key['snowflake-connector-python']._dep_map['pandas']
KeyError: KeyError('snowflake-connector-python',)

这是我的要求.txt

snowflake-connector-python
snowflake-sqlalchemy
sqlalchemy
pyarrow
pandas
asn1crypto==1.4.0
awscli==1.18.140
awswrangler==1.9.3
azure-common==1.1.25
azure-core==1.8.1
azure-storage-blob==12.5.0
boto3==1.14.63
botocore==1.17.63
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
colorama==0.4.3 ; python_version != '3.4'
cryptography==2.9.2
docutils==0.15.2
fsspec==0.8.2
idna==2.9
isodate==0.6.0
jmespath==0.10.0
msrest==0.6.19
numpy==1.19.2
oauthlib==3.1.0
oscrypto==1.2.1
packaging==20.4 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
psycopg2-binary==2.8.6
pyasn1==0.4.8
pycparser==2.20
pycryptodomex==3.9.8
pyjwt==1.7.1
pymysql==0.9.0
pyopenssl==19.1.0
pyparsing==2.4.7
python-dateutil==2.8.1
pytz==2020.1
pyyaml==5.3.1 ; python_version != '3.4'
requests-oauthlib==1.3.0
requests==2.23.0
rsa==4.5 ; python_version != '3.4'
s3fs==0.4.2
s3transfer==0.3.3
six==1.15.0
sqlalchemy-redshift==0.8.1
urllib3==1.25.10
xlrd==1.2.0

从日志文件中，这里是安装的包

Successfully installed asn1crypto-1.4.0 awscli-1.18.140 awswrangler-1.9.3 azure-common-1.1.25 azure-core-1.8.1 azure-storage-blob-12.5.0 boto3-1.14.63 botocore-1.17.63 certifi-2020.6.20 cffi-1.14.2 chardet-3.0.4 colorama-0.4.3 cryptography-2.9.2 dependencies-0.1.0 docutils-0.15.2 fsspec-0.8.2 idna-2.9 isodate-0.6.0 jmespath-0.10.0 msrest-0.6.19 numpy-1.19.2 oauthlib-3.1.0 oscrypto-1.2.1 packaging-20.4 pandas-1.1.2 psycopg2-binary-2.8.6 pyarrow-1.0.1 pyasn1-0.4.8 pycparser-2.20 pycryptodomex-3.9.8 pyjwt-1.7.1 pymysql-0.9.0 pyopenssl-19.1.0 pyparsing-2.4.7 python-dateutil-2.8.1 pytz-2020.1 pyyaml-5.3.1 requests-2.23.0 requests-oauthlib-1.3.0 rsa-4.5 s3fs-0.4.2 s3transfer-0.3.3 six-1.15.0 snowflake-connector-python-2.3.2 snowflake-sqlalchemy-1.2.3 sqlalchemy-1.3.19 sqlalchemy-redshift-0.8.1 urllib3-1.25.10 xlrd-1.2.0

【参考方案1】：

我建议您对所有主要库进行“干净”构建，其中包括 pyarrow 库。

例如

pip install --upgrade --force-reinstall pandas
pip install --upgrade --force-reinstall pyarrow
pip install --upgrade --force-reinstall snowflake-connector-python
pip install --upgrade --force-reinstall sqlalchemy
pip install --upgrade --force-reinstall snowflake-sqlalchemy

旧版本或缺少版本的 pyarrow 到处都存在一些问题，我相信即使安装顺序也可能是一个问题。全新的 pip 安装或确保您拥有这些库中最新/最好的库是一个不错的选择。

我希望这会有所帮助...丰富

附言如果这个（或另一个）答案对您有帮助，请花点时间“接受”有帮助的答案 通过单击答案旁边的复选标记将其从“灰色”切换为“已填充”。

【讨论】：

我在帖子中包含了 requirements.txt。我使用了一个 wheel 文件来包含这些包，然后上传到 s3 并在胶水作业中使用它。但我仍然得到同样的错误。