hadoop的日常操作

Posted 2021-10-09 醉舞斜陽

一、清理空间步骤

1.删除日志和回收站

hdfs dfs -rm -r -skipTrash /user/*/.Trash/*

hdfs dfs -rm -r -skipTrash /var/log/hadoop-yarn/apps/tools/logs/*

hdfs dfs -rm -r -skipTrash /group/user/*/meta/hive-exec/*/*

2.减少大表副本数

a、查找大表

hdfs dfs -du hdfs://dbdata/group/user/tools/meta/hive-temp-table hdfs://dbdata/group/user/tools/meta/hive-temp-table/comtysdktj hdfs://dbdata/group/user/tools/meta/hive-temp-table/kstj_os hdfs://dbdata/group/user/tools/meta/hive-temp-table/tysdktj hdfs://dbdata/group/user/tools/meta/hive-temp-table/otysdktj|awk \'{print $2 " " $3}\'|grep -v comtysdktj$|grep -v stj_os$|grep -v tysdktj$|grep -v otysdktj$|sort -k1,1nb|tail -100|awk \'{print $1/1024/1024/1024 "G " $2}\'

b、修改副本数

hadoop fs -setrep -w 1 -R hdfs://dbdata/group/user/tools/meta/hive-temp-table/s_tysdktj_db_home_nav_content_ys