所有查询的 Hive 错误:找不到与操作句柄关联的日志

Posted

技术标签:

【中文标题】所有查询的 Hive 错误:找不到与操作句柄关联的日志【英文标题】:Hive error on all queries: Couldn't find log associated with operation handle 【发布时间】:2015-08-19 08:59:49 【问题描述】:

在 Hadoop 上运行的所有 hive 查询都出现错误。

只需从存在的表中运行 select 语句并获得以下结果:

Fetching results ran into the following error(s):

Couldn't find log associated with operation handle: 
OperationHandle[opType=EXECUTE_STATEMENT,
getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]

查询的附带日志是:

请求 TFetchResultsReq(fetchType=1, operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='%\x1d\x0f\x11\xce\xf4B\xd6\xb8\xb5\x8ddv\xceKV', guid="\xbc5\xd9|\xfa\x13J'\x86\x8e4\xb3\x87\xa0\xf4\x05")), 方向=4,最大行数=-1): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage="不能 查找与操作句柄关联的日志:OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=bc35d97c-fa13-4a27-868e-34b387a0f405]", sqlState=无, infoMessages=["*org.apache.hive.service.cli.HiveSQLException: 不能 查找与操作句柄关联的日志:OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=bc35d97c-fa13-4a27-868e-34b387a0f405]:25:24", 'org.apache.hive.service.cli.operation.OperationManager:getOperationLogRowSet:OperationManager.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:656', 'sun.reflect.GeneratedMethodAccessor29:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy28:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:587', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:744'], statusCode=3), results=None, hasMoreRows=None) 请求 TFetchResultsReq(fetchType=1, operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='c\xdf\xfe\xb7\x8a\x10D\xea\xbaD\x93B\xb5\xc3\xe7', guid='&=\xf9\xc3\xc0vEO\xa0%#\xf5\xb9\x19\xe9W')), 方向=0, maxRows=-1): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage="找不到与操作句柄关联的日志: 操作句柄 [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]", sqlState=无, infoMessages=["*org.apache.hive.service.cli.HiveSQLException: 不能 查找与操作句柄关联的日志:OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]:25:24", 'org.apache.hive.service.cli.operation.OperationManager:getOperationLogRowSet:OperationManager.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:656', 'sun.reflect.GeneratedMethodAccessor29:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy28:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:587', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:744'], statusCode=3), results=None, hasMoreRows=None) 请求 TFetchResultsReq(fetchType=1, operationHandle=TOperationHandle(hasResultSet=False, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='c\xdf\xfe\xb7\x8a\x10D\xea\xbaD\x93B\xb5\xc3\xe7', guid='&=\xf9\xc3\xc0vEO\xa0%#\xf5\xb9\x19\xe9W')), 方向=0, maxRows=-1): TFetchResultsResp(status=TStatus(errorCode=0, errorMessage="找不到与操作句柄关联的日志: 操作句柄 [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]", sqlState=无, infoMessages=["*org.apache.hive.service.cli.HiveSQLException: 不能 查找与操作句柄关联的日志:OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=263df9c3-c076-454f-a025-23f5b919e957]:25:24", 'org.apache.hive.service.cli.operation.OperationManager:getOperationLogRowSet:OperationManager.java:257', 'org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:656', 'sun.reflect.GeneratedMethodAccessor29:invoke::-1', 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43', 'java.lang.reflect.Method:invoke:Method.java:606', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:79', 'org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:37', 'org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:64', 'java.security.AccessController:doPrivileged:AccessController.java:-2', 'javax.security.auth.Subject:doAs:Subject.java:415', 'org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1628', 'org.apache.hadoop.hive.shims.HadoopShimsSecure:doAs:HadoopShimsSecure.java:536', 'org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:60', 'com.sun.proxy.$Proxy28:fetchResults::-1', 'org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:427', 'org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:587', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1553', 'org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1538', 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39', 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39', 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56', 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:206', 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145', 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615', 'java.lang.Thread:run:Thread.java:744'], statusCode=3), results=None, hasMoreRows=None)

【问题讨论】:

【参考方案1】:

如果您在使用 Hue 时在 Cloudera 发行版中看到此问题,则将以下内容添加到 Cloudera Manager(或直接访问其他发行版的 hive-site.xml):

<property>
    <name>hive.server2.logging.operation.enabled</name>
    <value>true</value>
</property>

【讨论】:

非常感谢!非常适合我。【参考方案2】:

此错误可能发生在 即(“hive.server2.logging.operation.log.location”) 指向不存在写入日志的位置。

查看OperationManager.java

中的代码
OperationLog operationLog = getOperation(opHandle).getOperationLog();
if (operationLog == null) 
  throw new HiveSQLException("Couldn't find log associated with operation handle: " + opHandle);

【讨论】:

感谢您的回复。我将如何检查/更改它指向的位置? 点击此链接了解配置日志文件的位置。 cwiki.apache.org/confluence/display/Hive/… 通常这是由于连接字符串问题引起的,我也遇到过类似的问题,通常一旦我更正了我的连接字符串:jdbc://hive2::port/default,问题就会出现已解决。【参考方案3】:

如果您使用的是 AWS EMR,请将以下内容添加到位于 /etc/hive/conf.dist/hive-site.xml 的 hive-site.xml 文件中:

<property>
    <name>hive.server2.logging.operation.enabled</name>
    <value>true</value>
</property>

然后登录master节点重启hue和hive服务器

sudo stop hive-server2
sudo stop hue
sudo start hue
sudo start hive-server2

这为我修复了它(在发布标签 emr-4.6.0 中测试)。

【讨论】:

【参考方案4】:

如果您由于 shell 脚本或 java 程序而遇到此问题,您需要检查的第一件事是连接字符串,几个月前我遇到过类似的问题,当我在脚本中运行查询时以这种方式:

beeline -u  jdbc:hive2:// -e "show databases;"

这是由于缺少正确的 Hive 连接字符串,一旦我更正了连接字符串并运行查询,它就可以正常运行:

beeline -u  jdbc:hive2://server.enterprise.org:10000/default -e "show databases;"

此答案仅针对使用脚本或代码的开发人员,对于 Hue/Cloudera 发行版,请参考上述答案。

干杯!

【讨论】:

【参考方案5】:

在 Windows 8.1 中,我在 hive 查询部分中也面临同样的问题部分 -

org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated with operation handle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=230485c4-a1be-4a17-94c8-a1ebce426319]

因此,我去了%HIVE_HOME%/conf/hive-site.xml并将属性名称hive.server2.logging.operation.log.location之前的值$system:user.name/operation_logs更改为如下所述-

  <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/tmp/$user.name/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

并且还将包含$system:user.name 的所有值替换为/tmp/$user.name,这让我想到了hive 中的作业记录问题,谢谢。

【讨论】:

【参考方案6】:

在 Azure 中你可以使用

    beeline -u 'jdbc:hive2://headnodehost:10001/;transportMode=http'

【讨论】:

【参考方案7】:

创建表时使用以下代码 -

Create external table if not exists TEST1 
(
col1 string,
col2 int,
col3 date
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ('field.delim'=',') 
location 'hdfs:/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
tblproperties ("skip.header.line.count"="1")
;

【讨论】:

【参考方案8】:

选择是否使用 Hive 0.14 之前的旧 GetLog() thrift 调用来检索日志。 如果为 false,请改用 Hive 1.0 或更高版本的 FetchResults() thrift 调用

use_get_log_api=false

真为假

【讨论】:

以上是关于所有查询的 Hive 错误:找不到与操作句柄关联的日志的主要内容,如果未能解决你的问题,请参考以下文章

Spark访问与HBase关联的Hive表

Hybris:找不到与当前 URL 关联的 CMSSite

颤振打开 Hive 框会给出错误 - 说找不到框,您是不是忘记调用 Hive openbox()

HIVE:UDF 错误失败:找不到类 <ClassName>

Hive SQL Join关联查询

查询操作中的 BigQuery 错误:找不到项目 ID