无法使用 Zookeeper 连接字符串通过 JDBC 连接到 Hive

Posted 2023-02-16

【中文标题】无法使用 Zookeeper 连接字符串通过 JDBC 连接到 Hive

【英文标题】：Can't connect to Hive via JDBC using Zookeeper connection string

【发布时间】：2018-05-23 16:21:18

【问题描述】：

val driverClassName = "org.apache.hive.jdbc.HiveDriver";
Class.forName(driverClassName);
// The below connection string works
// val jdbcUrl = String.format("jdbc:hive2://sandbox-hdp.hortonworks.com:10000/%s", database);
// The next one doesn't...
val jdbcUrl = String.format("jdbc:hive2://sandbox-hdp.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2", database);
val jdbcUsername = "amy_ds";
val jdbcPassword = "amy_ds";
return DriverManager.getConnection(jdbcUrl, jdbcUsername, jdbcPassword);

我正在使用上述代码建立与 Hortonworks Data Platform Sandbox 2.6.4 打包的 Hive 实例的 JDBC 连接。如果我使用第一个（已注释掉的）连接字符串进行连接，则该连接已建立并可稍后在该方法中使用。但是，如果我尝试通过 Zookeeper 进行连接，则会引发异常：

2018-05-23 10:14:08 INFO  Utils:310 - Supplied authorities: sandbox-hdp.hortonworks.com:2181
2018-05-23 10:14:08 INFO  CuratorFrameworkImpl:230 - Starting
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:host.name=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_172
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:java.home=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:java.class.path=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:os.name=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:os.version=10.0
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:user.name=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:user.home=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:100 - Client environment:user.dir=[OMITTED FOR SECURITY]
2018-05-23 10:14:13 INFO  ZooKeeper:438 - Initiating client connection, connectString=sandbox-hdp.hortonworks.com:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@17935b13
2018-05-23 10:14:13 INFO  ClientCnxn:975 - Opening socket connection to server sandbox-hdp.hortonworks.com/[OMITTED FOR SECURITY]:2181. Will not attempt to authenticate using SASL (unknown error)
2018-05-23 10:14:13 INFO  ClientCnxn:852 - Socket connection established to sandbox-hdp.hortonworks.com/[OMITTED FOR SECURITY]:2181, initiating session
2018-05-23 10:14:13 INFO  ClientCnxn:1235 - Session establishment complete on server sandbox-hdp.hortonworks.com/[OMITTED FOR SECURITY]:2181, sessionid = 0x163891b08d300ed, negotiated timeout = 60000
2018-05-23 10:14:13 INFO  ConnectionStateManager:228 - State change: CONNECTED
2018-05-23 10:14:14 INFO  ZooKeeperHiveClientHelper:83 - Selected HiveServer2 instance with uri: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=sandbox-hdp.hortonworks.com;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2018-05-23 10:14:14 INFO  ZooKeeper:684 - Session: 0x163891b08d300ed closed
2018-05-23 10:14:14 INFO  ClientCnxn:512 - EventThread shut down
2018-05-23 10:14:14 INFO  Utils:397 - Resolved authority: hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=sandbox-hdp.hortonworks.com;hive.server2.thrift.port=10000;hive.server2.use.SSL=false
2018-05-23 10:14:14 INFO  HiveConnection:203 - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.server2.authentication=NONE;hive.server2.transport.mode=binary;hive.server2.thrift.sasl.qop=auth;hive.server2.thrift.bind.host=sandbox-hdp.hortonworks.com;hive.server2.thrift.port=10000;hive.server2.use.SSL=false/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
May 23, 2018 10:14:14 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause
java.lang.NullPointerException
    at org.apache.thrift.transport.TSocket.open(TSocket.java:170)
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176)
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at mil.navy.navair.ecbm.widgets.example.TestController.getHiveJdbcConnection(TestController.java:59)
    at mil.navy.navair.ecbm.widgets.example.TestController.allRecords(TestController.java:30)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

有什么想法吗？我看到的大多数文献都说要么更改 JDBC 驱动程序版本，要么尝试不同的连接字符串格式。我尝试了 1.2.1、2.3.0 和 3.0.0，但都没有工作（除非我使用已知的工作连接字符串直接连接到 Hive）。

我怀疑 JDBC 驱动程序在解释 Zookeeper 返回的 URL 时遇到问题，因为它说如果我尝试使用 Zookeeper 从日志中返回的 URL，它就找不到主机。

【问题讨论】：

同样的问题，如果你有答案，可以在这里发布答案吗？

【参考方案1】：

我遇到了同样的问题，正在努力解决它。发现 jdbc url 无法处理分号，尝试在端点和端口之后添加引号。

jdbc:hive2://sandbox-hdp.hortonworks.com:2181/';serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2'

这解决了我的问题。