大数据之---hadoop常用命令大全终极篇

Posted 2020-11-05

软件环境

RHEL6.8	hadoop2.8.1	apache-maven-3.3.9
findbugs-1.3.9	protobuf-2.5.0.tar.gz	jdk-8u45

(操作环境root安装启动的hadoop)
hadoop fs == hdfs dfs

将文件上传至hadoop的根目录/下载至本地
hadoop dfs -put filename /????
hadoop dfs -get /filename???
# ‘/‘不是Linux的根目录，表示hadoop的根目录
?
上传文件?? 下载
[[email protected] software]# hdfs dfs -put apache-maven-3.3.9-bin.zip? /
[[email protected] software]# hdfs? dfs -ls?? /
[[email protected] software]# hdfs dfs -put apache-maven-3.3.9-bin.zip? /
[[email protected] software]# cd
[[email protected] ~]# hdfs dfs -get /apache-maven-3.3.9-bin.zip

帮助

? [-moveFromLocal <localsrc> ... <dst>]
? [-moveToLocal <src> <localdst>]
查看hadoop里的文件内容
hadoop dfs -ls /
[[email protected] ~]# hdfs dfs -ls /
Found 3 items
-rw-r--r--?? 3 root supergroup??? 8617253 2018-04-26 09:20 /apache-maven-3.3.9-bin.zip
-rw-r--r--?? 3 root supergroup????? 59776 2018-04-26 09:14 /install.log
-rw-r--r--?? 3 root supergroup????? 11067 2018-04-26 08:46 /install.log.syslog
查看hadoop里的文件的内容
hadoop fs -cat filename
[[email protected] ~]# hadoop fs -cat?? /file1
111

创建文件夹

? hadoop fs -mkdir -p /filename/filename
[[email protected] ~]#?? hadoop fs -mkdir -p /filename/filename

删除file 删除dir
[[email protected] ~]# hadoop fs -rm -r -f /file1
Deleted /file1
[[email protected] ~]# hadoop fs -rm -r -f /filename
Deleted /filename

? #? [-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]
?
? 1.配置回收站
? vi core-site.xml
? <property>?
??? <name>fs.trash.interval</name>?
??? <value>10080</value>????????? # 回收站保留时间（分钟）
? </property>
?
?
? 2.测试
? hadoop fs -rm -r -f /xxxx????????????????????????????? 进入回收站，是可以恢复的
? hadoop fs -rm -r -f -skipTrash /xxxx?????? 不进入回收站，是不可以恢复的
? A.进入回收站
[[email protected] ~]# hadoop fs -ls /
Found 4 items
-rw-r--r--?? 3 root supergroup??? 8617253 2018-04-26 09:20 /apache-maven-3.3.9-bin.zip
drwxr-xr-x?? - root supergroup????????? 0 2018-04-26 09:35 /dir
-rw-r--r--?? 3 root supergroup????? 59776 2018-04-26 09:14 /install.log
-rw-r--r--?? 3 root supergroup????? 11067 2018-04-26 08:46 /install.log.syslog
[[email protected] ~]# hadoop fs -rm -r -f /install.log
18/04/26 09:38:28 INFO fs.TrashPolicyDefault: Moved: ‘hdfs://172.16.18.133:9000/install.log‘ to trash at: hdfs://172.16.18.133:9000/user/root/.Trash/Current/install.log
[[email protected] ~]# hadoop fs -ls
Found 1 items
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 .Trash
[[email protected] ~]# hadoop fs -ls
Found 1 items
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 .Trash
[[email protected] ~]# hadoop fs -ls .Trash/
Found 1 items
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 .Trash/Current
[[email protected] ~]# hadoop fs -ls .Trash/Current
Found 1 items
-rw-r--r--?? 3 root supergroup????? 59776 2018-04-26 09:14 .Trash/Current/install.log
? 恢复刚刚删除的文件
[[email protected] ~]# hadoop fs -ls /user
Found 1 items
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 /user/root
[[email protected] ~]# hadoop fs -ls /user/root/
Found 1 items
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 /user/root/.Trash
[[email protected] ~]# hadoop fs -mv /user/root/.Trash/Current/install.log? /
[[email protected] ~]# hadoop fs -ls /
Found 5 items
-rw-r--r--?? 3 root supergroup??? 8617253 2018-04-26 09:20 /apache-maven-3.3.9-bin.zip
drwxr-xr-x?? - root supergroup????????? 0 2018-04-26 09:35 /dir
-rw-r--r--?? 3 root supergroup????? 59776 2018-04-26 09:14 /install.log
-rw-r--r--?? 3 root supergroup????? 11067 2018-04-26 08:46 /install.log.syslog
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 /user
?
? B.不进入回收站
[[email protected] ~]# hadoop fs -rm -r -f -skipTrash? /install.log.syslog
Deleted /install.log.syslog
[[email protected] ~]# hadoop fs -ls
Found 1 items
drwx------?? - root supergroup????????? 0 2018-04-26 09:38 .Trash
[[email protected] ~]# hadoop fs -ls .Trash/Current
[[email protected] ~]#????????

hdfs fs -moveFromLocal /input/xx.txt /input/xx.txt? ---从本地剪切粘贴到hdfs

hdfs fs -moveToLocal /input/xx.txt /input/xx.txt? --从hdfs剪切粘贴到本地

hdfs fs -appedToFile ./hello.txt /input/hello.txt? ---追加一个文件到另一个文件到末尾

注意：appedToFile? moveToLocal moveFromLocal? 大小写严格

hdfs fs -cat /input/hello.txt?? ---查看文件内容

hdfs fs -tail /input/hello.txt?? ---显示一个文件到末尾

hdfs fs -text /input/hello.txt? ---以字符串的形式打印文件的内容

hdfs fs -chmod 666 /input/hello.txt?? ---修改文件权限

hdfs fs -chown ruoze.ruoze? /input/hello.txt?? --修改文件所属

hdfs fs -copyFromLocal /input/hello.txt /input/?? --从本地文件系统拷贝到hdfs里

hdfs fs -copyToLocal /input/hello.txt /input/???? --从hdfs拷贝到本地

hdfs fs -cp /input/xx.txt /output/xx.txt????????????? ---从hdfs到一个路径拷贝到另一个路径

hdfs fs -mv /input/xx.txt /output/xx.txt??????????? ---从hdfs到一个路径移动到另一个路径

hdfs fs -df -h /????????????????????????????????????????????????????????????? ----统计文件系统的可用空间信息

hdfs fs -du -s -h /???????????????????????????????????????????????????????? ----统计文件夹的大小信息

hadoop? fs -count /aaa? -???????????????????????????????????????????? ---统计一个指定目录下的文件节点数量

hadoop fs -setrep 3 /input/xx.txt?????????????????????????? --设置hdfs的文件副本数量