来自 Presto 数据库的 Sqoop，[Teradata][JDBC](11975) 不支持的事务隔离级别：2

Posted 2023-03-21

技术标签:

【中文标题】来自 Presto 数据库的 Sqoop，[Teradata][JDBC](11975) 不支持的事务隔离级别：2【英文标题】：Sqoop from Presto Database, [Teradata][JDBC](11975) Unsupported transaction isolation level: 2 【发布时间】：2018-08-05 11:16:13 【问题描述】：

我可以使用 Cli、Spark 等连接和查询 Presto 数据库

但是当我尝试将 PrestoDB 实例中的 Sqoop 导入 S3 时，我收到了这个事务隔离错误：

    8/02/26 03:03:46 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
18/02/26 03:03:46 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
18/02/26 03:03:46 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
18/02/26 03:03:46 INFO manager.SqlManager: Using default fetchSize of 1000
18/02/26 03:03:46 INFO tool.CodeGenTool: Beginning code generation
18/02/26 03:03:46 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM alarm_types AS t WHERE 1=0
18/02/26 03:03:46 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
18/02/26 03:05:56 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: [Teradata][JDBC](11975) Unsupported transaction isolation level: 2.
java.sql.SQLException: [Teradata][JDBC](11975) Unsupported transaction isolation level: 2.
    at com.teradata.exceptions.ExceptionConverter.toSQLException(Unknown Source)
    at com.teradata.jdbc.common.SConnection.setTransactionIsolation(Unknown Source)
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:891)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
    at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
18/02/26 03:05:56 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

我尝试使用标准 Facebook 驱动程序 jar 和 Teradata Presto Jdbc jar，这两个驱动程序都出现相同的错误。

使用 Facebook 驱动时的异常：

    18/02/26 19:22:20 ERROR manager.SqlManager: Error executing statement: java.sql.SQLFeatureNotSupportedException: Transactions are not yet supported
java.sql.SQLFeatureNotSupportedException: Transactions are not yet supported
    at com.facebook.presto.jdbc.PrestoConnection.setTransactionIsolation(PrestoConnection.java:220)
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:891)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:744)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:767)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
    at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
18/02/26 19:22:20 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
    at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
    at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

这是我的 Sqoop 命令：

sqoop import \
    -libjars $teradata_jar_path --driver $teradata_driver \
    --verbose \
    --connect jdbc:presto://xxx/<db> \
    --table <tablename> --target-dir s3://<s3-path> \
     --relaxed-isolation

请注意，我也尝试使用“--relaxed-isolation”，但得到相同的错误。任何帮助将不胜感激。

谢谢

【问题讨论】：

能否请您在使用 Facebook 驱动程序时发布堆栈跟踪信息？感谢@kokosing，当使用 facebook 驱动程序时，我用 stacktrace 更新了我的问题。有什么线索吗？您使用的是什么版本的 Presto。看到支持 0.195 事务：github.com/prestodb/presto/blob/master/presto-jdbc/src/main/… 我目前正在使用 Presto 版本：0.179-T.1 和 presto-jdbc-0.184.jar 驱动程序。感谢您对 0.195 版本的提示，会检查一下。切换到 0.195 后，我能够克服事务问题，但现在得到 com.facebook.presto.jdbc.NotImplementedException: Method Connection.prepareStatement is not yet implemented 【参考方案1】：

请升级您的 Presto 安装。我建议你使用最新的。

注意在 0.189 中添加了对使用 SQL 语句或标准 JDBC 机制的事务的支持：https://prestosql.io/docs/current/release/release-0.189.html

【讨论】：

感谢 Kokosing 的建议，我们生产的 Presto 版本目前是 0.179，我建议升级。嗨@kokosing，我尝试使用 Facebook Presto 驱动程序版本：presto-jdbc-0.205.jar 但仍然出现错误：18/07/12 10:47:02 INFO manager.SqlManager: Executing SQL statement : SELECT t.* FROM hive.thor_fttx.v_ams_inv_ont_optical_measurements AS t WHERE 1=0 18/07/12 10:47:02 ERROR manager.SqlManager: Error execution statement: java.sql.SQLException: Query failed (#20180712_104701_01127_ptpsh): Connector支持的隔离级别 READ UNCOMMITTED 不符合请求的隔离级别 READ COMMITTED

以上是关于来自 Presto 数据库的 Sqoop，[Teradata][JDBC](11975) 不支持的事务隔离级别：2的主要内容，如果未能解决你的问题，请参考以下文章

使用来自 Mysql 的 sqoop 导入 Hive 耗时太长

使用 Sqoop 将来自 MySQL 的数据加载到 PIG

Prestissimo：使 Presto 性能提升三倍

Presto 数组包含来自另一列的值（超集 SQL 查询）

Presto架构及原理