hive归档分区

Posted ChavinKing

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了hive归档分区相关的知识,希望对你有一定的参考价值。

归档hive历史分区不会减少hdfs存储空间,但是可以有效减轻hadoop namenode的压力,尤其在于小文件比较多的情况下。

$mkdir $HIVE_HOME/auxlib

$ cp /opt/cdh-5.3.6/hadoop-2.5.0/share/hadoop/tools/lib/hadoop-archives-2.5.0-cdh5.3.6.jar /opt/cdh-5.3.6/hive-0.13.1/auxlib/hadoop-archives-2.5.0-cdh5.3.6.jar

 

hive (chavin)>  set hive.archive.enabled=true;                                    
hive (chavin)> alter table emp archive partition(country=\'china\',state=\'beijing\');
intermediate.archived is hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ARCHIVED
intermediate.original is hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ORIGINAL
Creating data.har for hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing
in hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing/.hive-staging_hive_2017-05-24_13-23-51_914_3548751700804069937-1/-ext-10000/partlevel
Please wait... (this may take a while)
Moving hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing/.hive-staging_hive_2017-05-24_13-23-51_914_3548751700804069937-1/-ext-10000/partlevel to hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ARCHIVED
Moving hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing to hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ORIGINAL
Moving hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ARCHIVED to hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing
Moved: \'hdfs://db01:8020/user/hive/warehouse/chavin.db/emp/country=china/state=beijing_INTERMEDIATE_ORIGINAL\' to trash at: hdfs://db01:8020/user/hadoop/.Trash/Current
OK
Time taken: 2.17 seconds

 

可以看到,emp表下分区文件已经被打包成data.har文件了。

 

image

以上是关于hive归档分区的主要内容,如果未能解决你的问题,请参考以下文章

hive 分区表

RDS for MySQL 通过分区归档历史数据

hive创建分区 9870查询不到

hive的严格模式和分区

hive外部表分区

hive中怎样删除分区