hiveserver2 org.apache.thrift.transport.TTransportException 在不活动分钟后运行第二个查询时出错

Posted 2023-04-18

【中文标题】hiveserver2 org.apache.thrift.transport.TTransportException 在不活动分钟后运行第二个查询时出错

【英文标题】：hiveserver2 org.apache.thrift.transport.TTransportException error when running 2nd query after minute of inactivity

【发布时间】：2018-04-05 00:48:44

【问题描述】：

我使用 10000 端口从我的 SQL 工具（即 Squirrel SQL、Oracle SQL Developer）到 HiveServer2（在远程服务器上运行）建立 JDBC 连接。

我能够成功运行一些查询。 然后我执行其他操作（不在 SQL 工具中）1-2 分钟，然后返回到我的 SQL 工具并尝试运行查询，但出现此错误：org.apache.thrift.transport.TTransportException: java.net.SocketException: Software caused connection abort: socket write error

如果我现在在我的 SQL 工具中断开连接并重新连接，我可以再次运行查询。但是有谁知道我应该更改哪些 HiveServer2 设置以防止出现错误？我假设 hive-site.xml 中有一些东西

从下面的 hiveserver2 日志中，可以看到从第 30 分钟到第 31 分钟发生断开连接的确切 1 分钟间隔。

2018-04-05T03:30:41,706  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:30:41,712  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:30:41,712  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:30:41,718  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:30:41,719  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:31:41,232  INFO [HiveServer2-Handler-Pool: Thread-36] thrift.ThriftCLIService: Session disconnected without closing properly.
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] thrift.ThriftCLIService: Closing the session: SessionHandle [c81ec0f9-7a9d-46b6-9708-e7d78520a48a]
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] service.CompositeService: Session closed, SessionHandle [c81ec0f9-7a9d-46b6-9708-e7d78520a48a], current sessions:0
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Updating thread name to c81ec0f9-7a9d-46b6-9708-e7d78520a48a HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] session.HiveSessionImpl: Operation log session directory is deleted: /var/hive/hs2log/tmp/c81ec0f9-7a9d-46b6-9708-e7d78520a48a
2018-04-05T03:31:41,233  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-36
2018-04-05T03:31:41,236  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Deleted directory: /var/hive/scratch/tmp/anonymous/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with scheme file
2018-04-05T03:31:41,236  INFO [HiveServer2-Handler-Pool: Thread-36] session.SessionState: Deleted directory: /var/hive/ec2-user/c81ec0f9-7a9d-46b6-9708-e7d78520a48a on fs with scheme file
2018-04-05T03:31:41,236  INFO [HiveServer2-Handler-Pool: Thread-36] hive.metastore: Closed a connection to metastore, current connections: 1

环境：

Hive 2.1.1，hive.server2.transport.mode 设置为二进制（示例 JDBC 字符串为 jdbc:hive2://remotehost:10000/default） Hadoop 2.8.3 使用 mysql 的元存储 Java 8

【参考方案1】：

将 AWS ELB 的默认超时时间从 1 分钟更改为 20 分钟 将 SimbaHive 驱动程序与 Squirrel 一起使用（无超时）或将标准 apache hive2 驱动程序与 IBM Data Studio 一起使用（无超时）