记一次hive查询报错解决 No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP

Posted 2022-12-03 扫地增

背景

在正常开发中建表进行数据插入测试，突然报错
代码如下：

INSERT OVERWRITE TABLE dim_common.dim_common_product_..._sku
SELECT 
   id,
  .....
FROM ods_common.ods_common_product_..._sku

建表语句：

CREATE EXTERNAL TABLE dim_common.dim_common_product_..._sku(
  `id` bigint COMMENT '主键',
......)
COMMENT 
STORED AS parquet
LOCATION '/big-data/dim/common/dim_common_product_..._sku'
TBLPROPERTIES ('parquet.compression'='lzop');

报错如下

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:211)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:742)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1260)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:733)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:757)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:193)
	... 9 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:285)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:780)
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:731)
	... 15 more
Caused by: java.lang.IllegalArgumentException: No enum constant org.apache.parquet.hadoop.metadata.CompressionCodecName.LZOP
	at java.lang.Enum.valueOf(Enum.java:238)
	at org.apache.parquet.hadoop.metadata.CompressionCodecName.valueOf(CompressionCodecName.java:26)
	at org.apache.parquet.hadoop.metadata.CompressionCodecName.fromConf(CompressionCodecName.java:39)
	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.initializeSerProperties(ParquetRecordWriterWrapper.java:119)
	at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:65)
	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:137)
	at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:126)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
	... 17 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 2  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

报错分析：

首先去掉了insert语句测试了查询语句发现正常执行，排除了查询语句错误。
添加insert之后报上错误，因此推测是集群压缩配置或者建表压缩配置有问题，因为并不是第一次建表，其他表均无异常。所以详细检查了建表语句最终发现压缩配置错误。
错误配置：

TBLPROPERTIES ('parquet.compression'='lzop');

修改为正确配置后解决：

TBLPROPERTIES ('parquet.compression'='lzo');

这里仅此小记，总结粗心教训，为大家解决问题提供思路。