Hive JDBC 连接超时

Posted 2023-03-25
技术标签:
【中文标题】Hive JDBC 连接超时【英文标题】：hive jdbc connection time out 【发布时间】：2017-03-28 02:25:49 【问题描述】：
我在服务器 node25 上安装 hive2.2，并尝试使用 jdbc 访问 hiveserver2 并使用单个查询测试“select * from mytest limit 5”，大多数情况下会导致异常： ..... 引起：java.net.ConnectException：连接超时：连接在 org.apache.thrift.transport.TSocket.open(TSocket.java:221) 它偶尔会成功，但使用直线没有问题。这是我的 hiveserver2 日志：
2017-03-28T09:59:28,347 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslServerTransport: transport map does not contain key
2017-03-28T09:59:28,347 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: opening transport org.apache.thrift.transport.TSaslServerTransport@4568f03
2017-03-28T09:59:28,464 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: Received message with status START and payload length 5
2017-03-28T09:59:28,464 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslServerTransport: Received start message with status START
2017-03-28T09:59:28,464 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslServerTransport: Received mechanism name 'PLAIN'
2017-03-28T09:59:28,465 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: Start message handled
2017-03-28T09:59:28,465 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: Received message with status COMPLETE and payload length 10
2017-03-28T09:59:28,569 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: Writing message with status COMPLETE and payload length 0
2017-03-28T09:59:28,569 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: Main negotiation loop complete
2017-03-28T09:59:28,570 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslServerTransport: transport map does contain key org.apache.thrift.transport.TSocket@a916021
2017-03-28T09:59:28,858 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: reading data length: 114
2017-03-28T09:59:28,858  INFO [HiveServer2-Handler-Pool: Thread-133] thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V9
2017-03-28T09:59:28,858 DEBUG [HiveServer2-Handler-Pool: Thread-133] thrift.ThriftCLIService: Client's IP Address: 192.168.190.168
2017-03-28T09:59:28,858 DEBUG [HiveServer2-Handler-Pool: Thread-133] thrift.ThriftCLIService: Client's username: hive
2017-03-28T09:59:28,858 DEBUG [HiveServer2-Handler-Pool: Thread-133] thrift.ThriftCLIService: Client's IP Address: 192.168.190.168
2017-03-28T09:59:28,860 DEBUG [HiveServer2-Handler-Pool: Thread-133] session.SessionState: SessionState user: hive
2017-03-28T09:59:28,862 DEBUG [HiveServer2-Handler-Pool: Thread-133] hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
2017-03-28T09:59:28,862 DEBUG [HiveServer2-Handler-Pool: Thread-133] hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
2017-03-28T09:59:28,862 DEBUG [HiveServer2-Handler-Pool: Thread-133] hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
2017-03-28T09:59:28,862 DEBUG [HiveServer2-Handler-Pool: Thread-133] hdfs.BlockReaderLocal: dfs.domain.socket.path = 
2017-03-28T09:59:28,863 DEBUG [HiveServer2-Handler-Pool: Thread-133] retry.RetryUtils: multipleLinearRandomRetry = null
2017-03-28T09:59:28,864 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@78d39a69
2017-03-28T09:59:28,864 DEBUG [HiveServer2-Handler-Pool: Thread-133] sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
2017-03-28T09:59:28,865 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.Client: The ping interval is 60000 ms.
2017-03-28T09:59:28,865 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.Client: Connecting to node25/192.168.156.25:9000
2017-03-28T09:59:28,866 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive: starting, having connections 1
2017-03-28T09:59:28,868 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #47
2017-03-28T09:59:28,883 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #47
2017-03-28T09:59:28,883 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: getFileInfo took 18ms
2017-03-28T09:59:28,884 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #48
2017-03-28T09:59:28,885 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #48
2017-03-28T09:59:28,886 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: getFileInfo took 2ms
2017-03-28T09:59:28,886 DEBUG [HiveServer2-Handler-Pool: Thread-133] session.SessionState: HDFS root scratch dir: /tmp/hive with schema null, permission: rwx-wx-wx
2017-03-28T09:59:28,886 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #49
2017-03-28T09:59:28,887 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #49
2017-03-28T09:59:28,888 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
2017-03-28T09:59:28,889 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #50
2017-03-28T09:59:28,890 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #50
2017-03-28T09:59:28,890 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
2017-03-28T09:59:28,890 DEBUG [HiveServer2-Handler-Pool: Thread-133] hdfs.DFSClient: /tmp/hive/hive/d67b7df6-60fc-483b-888e-94c08c430980: masked=rwx------
2017-03-28T09:59:28,891 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #51
2017-03-28T09:59:28,901 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #51
2017-03-28T09:59:28,901 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: mkdirs took 10ms
2017-03-28T09:59:28,901  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Created HDFS directory: /tmp/hive/hive/d67b7df6-60fc-483b-888e-94c08c430980
2017-03-28T09:59:28,902 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #52
2017-03-28T09:59:28,902 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #52
2017-03-28T09:59:28,903 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
2017-03-28T09:59:28,910  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Created local directory: /root/yuxh/app/apache-hive-2.2.0-SNAPSHOT-bin/scratchdir/d67b7df6-60fc-483b-888e-94c08c430980
2017-03-28T09:59:28,911 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #53
2017-03-28T09:59:28,912 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #53
2017-03-28T09:59:28,912 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms
2017-03-28T09:59:28,913 DEBUG [HiveServer2-Handler-Pool: Thread-133] hdfs.DFSClient: /tmp/hive/hive/d67b7df6-60fc-483b-888e-94c08c430980/_tmp_space.db: masked=rwx------
2017-03-28T09:59:28,913 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #54
2017-03-28T09:59:28,918 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #54
2017-03-28T09:59:28,919 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: mkdirs took 6ms
2017-03-28T09:59:28,919  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Created HDFS directory: /tmp/hive/hive/d67b7df6-60fc-483b-888e-94c08c430980/_tmp_space.db
2017-03-28T09:59:28,920 DEBUG [HiveServer2-Handler-Pool: Thread-133] session.HiveSessionImpl: Global init file /root/yuxh/app/apache-hive-2.2.0-SNAPSHOT-bin/apache-hive-2.2.0-SNAPSHOT-bin/conf does not exist
2017-03-28T09:59:28,920 DEBUG [HiveServer2-Handler-Pool: Thread-133] conf.VariableSubstitution: Substitution is on: hive
2017-03-28T09:59:28,920  INFO [HiveServer2-Handler-Pool: Thread-133] session.HiveSessionImpl: Operation log session directory is created: /root/yuxh/app/apache-hive-2.2.0-SNAPSHOT-bin/apache-hive-2.2.0-SNAPSHOT-bin/bin/$system:java.io.tmpdir/$system:user.name/operation_logs/d67b7df6-60fc-483b-888e-94c08c430980
2017-03-28T09:59:28,921  INFO [HiveServer2-Handler-Pool: Thread-133] service.CompositeService: Session opened, SessionHandle [d67b7df6-60fc-483b-888e-94c08c430980], current sessions:2
2017-03-28T09:59:28,921 DEBUG [HiveServer2-Handler-Pool: Thread-133] cli.CLIService: SessionHandle [d67b7df6-60fc-483b-888e-94c08c430980]: openSession()
2017-03-28T09:59:28,921 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: writing data length: 169
2017-03-28T09:59:38,914 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive: closed
2017-03-28T09:59:38,914 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive: stopped, remaining connections 0
2017-03-28T09:59:50,241 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: SERVER: reading data length: 83
2017-03-28T09:59:50,241  INFO [HiveServer2-Handler-Pool: Thread-133] service.CompositeService: Session closed, SessionHandle [d67b7df6-60fc-483b-888e-94c08c430980], current sessions:1
2017-03-28T09:59:50,242  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Updating thread name to d67b7df6-60fc-483b-888e-94c08c430980 HiveServer2-Handler-Pool: Thread-133
2017-03-28T09:59:50,243  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-133
2017-03-28T09:59:50,243  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Updating thread name to d67b7df6-60fc-483b-888e-94c08c430980 HiveServer2-Handler-Pool: Thread-133
2017-03-28T09:59:50,244  INFO [HiveServer2-Handler-Pool: Thread-133] session.HiveSessionImpl: Operation log session directory is deleted: /root/yuxh/app/apache-hive-2.2.0-SNAPSHOT-bin/apache-hive-2.2.0-SNAPSHOT-bin/bin/$system:java.io.tmpdir/$system:user.name/operation_logs/d67b7df6-60fc-483b-888e-94c08c430980
2017-03-28T09:59:50,244  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-133
2017-03-28T09:59:50,244 DEBUG [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Removing resource dir /root/yuxh/app/apache-hive-2.2.0-SNAPSHOT-bin/resources/d67b7df6-60fc-483b-888e-94c08c430980_resources
2017-03-28T09:59:50,245 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.Client: The ping interval is 60000 ms.
2017-03-28T09:59:50,246 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.Client: Connecting to node25/192.168.156.25:9000
2017-03-28T09:59:50,247 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive: starting, having connections 1
2017-03-28T09:59:50,247 DEBUG [IPC Parameter Sending Thread #4] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive sending #55
2017-03-28T09:59:50,259 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive got value #55
2017-03-28T09:59:50,260 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.ProtobufRpcEngine: Call: delete took 14ms
2017-03-28T09:59:50,260  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Deleted directory: /tmp/hive/hive/d67b7df6-60fc-483b-888e-94c08c430980 on fs with scheme hdfs
2017-03-28T09:59:50,261  INFO [HiveServer2-Handler-Pool: Thread-133] session.SessionState: Deleted directory: /root/yuxh/app/apache-hive-2.2.0-SNAPSHOT-bin/scratchdir/d67b7df6-60fc-483b-888e-94c08c430980 on fs with scheme file
2017-03-28T09:59:50,263 DEBUG [HiveServer2-Handler-Pool: Thread-133] metastore.ObjectStore: Cleared ClassLoaderResolverImpl: 0,0,0
2017-03-28T09:59:50,264 DEBUG [HiveServer2-Handler-Pool: Thread-133] metastore.ObjectStore: Removed cached classloaders from DataNucleus NucleusContext
2017-03-28T09:59:50,264 DEBUG [HiveServer2-Handler-Pool: Thread-133] metadata.Hive: Closing current thread's connection to Hive Metastore.
2017-03-28T09:59:50,264 DEBUG [HiveServer2-Handler-Pool: Thread-133] ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@78d39a69
2017-03-28T09:59:50,265 DEBUG [HiveServer2-Handler-Pool: Thread-133] cli.CLIService: SessionHandle [d67b7df6-60fc-483b-888e-94c08c430980]: closeSession()
2017-03-28T09:59:50,265 DEBUG [HiveServer2-Handler-Pool: Thread-133] transport.TSaslTransport: writing data length: 40
2017-03-28T10:00:00,249 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive: closed
2017-03-28T10:00:00,249 DEBUG [IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive] ipc.Client: IPC Client (1279469670) connection to node25/192.168.156.25:9000 from hive: stopped, remaining connections 0
好像连查询都没有触发，我觉得hive里面有一些配置需要设置。
【问题讨论】：
【参考方案1】：
我想肯定是网络问题（我的本地电脑用***连接集群网络），我在其中一个集群节点上运行程序就可以了。
【讨论】：
以上是关于Hive JDBC 连接超时的主要内容，如果未能解决你的问题，请参考以下文章
Derby/JDBC 连接生命周期（或空闲超时）
JDBC连接超时，针对连接不稳定，有时候能连上（登录），一会又报连接超时
JDBC 连接超时无法重新连接
JDBC 连接：超时后未重新连接
为啥@jdbc 查询因连接超时而失败？