将 csv 数据加载到表中使用 Hive SQL java 错误 InvocationTargetException

Posted 2023-03-23

【中文标题】将 csv 数据加载到表中使用 Hive SQL java 错误 InvocationTargetException

【英文标题】：Load csv data to table use Hive SQL java error InvocationTargetException

【发布时间】：2017-12-04 04:23:48

【问题描述】：

我疯了，因为无法使用 Hive SQL 将数据加载到表中。请教我如何修复这个错误

在元数据中创建表。我使用 2 语句：

1 - 创建表：

CREATE TABLE db1.test_table_file3(
Col1 INT,Col2 SMALLINT,Col3 SMALLINT)
COMMENT 'test_table_file2'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'

2 - 将数据加载到表中：

LOAD DATA INPATH '/test/test_test_5.csv' INTO TABLE 
db1.test_table_file3

代码如下：

Connection con = getConnection();
Statement stmt = con.createStatement();
Statement loadDataStmt = con.createStatement();
stmt.execute(createTableQuery);
loadDataStmt .execute(loadDataSQL);
closeConnection(con, stmt);
closeConnection(con, loadDataStmt);

结果如下：

Statement 1 create table: Execute successful  
Statement 2 Load data: Execute fail  

java.sql.SQLException: java.lang.reflect.InvocationTargetException
    at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:296) ~[hive-jdbc-1.2.1.spark2.jar:1.2.1.spark2]
    at com.toshiba.mwcloud.datalake.griddatalake.repository.impl.MetadataRepositoryImpl.loadDataIntoTable(MetadataRepositoryImpl.java:404) ~[bin/:na]
    at com.toshiba.mwcloud.datalake.griddatalake.repository.impl.MetadataRepositoryImpl.createMetadata(MetadataRepositoryImpl.java:49) ~[bin/:na]
    at com.toshiba.mwcloud.datalake.griddatalake.repository.impl.MetadataRepositoryImpl$$FastClassBySpringCGLIB$$10d6b423.invoke(<generated>) ~[bin/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:747) ~[spring-aop-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) ~[spring-aop-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at com.toshiba.mwcloud.datalake.griddatalake.repository.impl.MetadataRepositoryImpl$$EnhancerBySpringCGLIB$$bba82441.createMetadata(<generated>) ~[bin/:na]
    at com.toshiba.mwcloud.datalake.griddatalake.service.impl.MetadataServiceImpl.createMetadata(MetadataServiceImpl.java:85) ~[bin/:na]
    at com.toshiba.mwcloud.datalake.griddatalake.controller.MetadataController.createTable(MetadataController.java:117) ~[bin/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:871) ~[spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:777) ~[spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) [spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) [spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) [spring-webmvc-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-embed-websocket-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-5.0.2.BUILD-SNAPSHOT.jar:5.0.2.BUILD-SNAPSHOT]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.23.jar:8.5.23]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

【问题讨论】：

您是否尝试在 Java / Spring 之外运行这些命令？

@cricket_007 是的，我在命令中运行没问题

【参考方案1】：

问题在第二个语句，下面是它的解释。

如果您在 HDFS 中有数据，那么LOAD DATA INPATH 'hdfs:/test/test_test_5.csv' INTO TABLE db1.test_table_file3 将起作用。

如果你有本地文件，你应该使用 LOCAL INPATH LOAD DATA LOCAL INPATH '/test/test_test_5.csv' INTO TABLE db1.test_table_file3

如果您从 HDFS 加载文件，请在文件路径前提及名称节点

我建议您先从 JAVA 中运行第二条语句，看看您遇到了什么错误。人们通常会遇到外部表的权限问题，请注意 HIVE 用户应该有权访问 HDFS 路径。

你可以参考this页面，很简单的解释。

我注意到您的问题中的另一个问题，您在表格中将字段分隔符设置为制表符“\t”，并且您正在尝试插入 CSV 文件，不确定这是否是您的问题示例中的错误.

如果您遇到一些错误，请对错误进行评论，我将能够为您提供帮助。

【讨论】：

非常感谢@rob 关于您的评论，我尝试了 LOAD DATA INPATH 和 LOAD DATA LOCAL INPATH 但它似乎仍然不是这个错误的原因。使用 DATA 或 DATA LOCAL 错误始终是 InvocationTargetException。可能是我的配置不允许将数据加载到 HDFS 中的表中。您关于制表符的注意事项是正确的，它必须是逗号字符。

@Nguyen.Giang.93 如果您有访问权限，为什么不从 cmd 行尝试，这将为您提供很好的验证测试。还可以尝试将表更改为外部表并将表的位置设置为文件位置并查看。

是的，我在 cmd 中测试没问题。 cmd建表加载数据成功，java执行失败。