大数据讲课笔记3.4 Hadoop集群测试

Posted 2022-10-05 howard2005

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了大数据讲课笔记3.4 Hadoop集群测试相关的知识，希望对你有一定的参考价值。

文章目录

零、学习目标
一、导入新课
二、新课讲解
三、归纳总结
四、上机操作

零、学习目标

掌握通过UI界面查看Hadoop集群运行状态
熟悉Hadoop集群初体验的操作——词频统计

一、导入新课

上次课对Hadoop集群配置、格式化文件系统以及启动和关闭Hadoop集群进行了详细的讲解，本次课将针对通过UI界面查看Hadoop集群运行状态和Hadoop集群初体验进行详细讲解。

二、新课讲解

（一）通过UI界面查看Hadoop运行状态

Hadoop集群正常启动后，它默认开放了两个端口9870和8088，分别用于监控HDFS集群和YARN集群。通过UI界面可以方便地进行集群的管理和查看，只需要在本地操作系统的浏览器输入集群服务的IP和对应的端口号即可访问。

1、hadoop2和hadoop3端口区别表

2、查看HDFS集群状态

在浏览器里访问http://master:9870
不能通过主机名master加端口9870的方式，原因在于没有在hosts文件里IP与主机名的映射，现在只能通过IP地址加端口号的方式访问：http://192.168.1.101:9870
修改宿主机的C:\\Windows\\System32\\drivers\\etc\\hosts文件，增加hadoop集群主机名与IP地址的映射
此时，访问http://master:9870，从图中可以看出HDFS集群状态显示正常。
单击导航条上的【Datanodes】，查看数据节点信息
点开【Utilities】下拉菜单，选择【Browse the file system】
此时HDFS上什么东东都木有
在HDFS上创建一个目录BigData，执行命令：hdfs dfs -mkdir /BigData
在Hadoop WebUI界面查看刚才创建的目录

3、查看YARN集群状态

访问http://master:8088/cluster，从图中可以看出YARN集群状态显示正常。
单击[About]链接

（二）Hadoop集群初体验 —— 词频统计

1、启动Hadoop集群

在master虚拟机上执行命令：start-all.sh

2、在虚拟机上准备文件

在master虚拟机上创建test.txt文件

3、文件上传到HDFS指定目录

上传test.txt文件到HDFS的/BigData目录
利用HDFS命令查看文件是否上传成功
利用Hadoop WebUI查看文件是否上传成功

4、运行词频统计程序的jar包

查看Hadoop自带示例的jar包
执行命令：hadoop jar ./hadoop-mapreduce-examples-3.3.4.jar wordcount /BigData/test.txt /wc_result

[root@master mapreduce]# hadoop jar ./hadoop-mapreduce-examples-3.3.4.jar wordcount /BigData/test.txt /wc_result
2022-10-04 12:40:30,559 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at master/192.168.1.101:8032
2022-10-04 12:40:31,501 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1664901602347_0001
2022-10-04 12:40:32,416 INFO input.FileInputFormat: Total input files to process : 1
2022-10-04 12:40:32,911 INFO mapreduce.JobSubmitter: number of splits:1
2022-10-04 12:40:33,204 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1664901602347_0001
2022-10-04 12:40:33,205 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-10-04 12:40:33,418 INFO conf.Configuration: resource-types.xml not found
2022-10-04 12:40:33,418 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-10-04 12:40:33,939 INFO impl.YarnClientImpl: Submitted application application_1664901602347_0001
2022-10-04 12:40:33,974 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1664901602347_0001/
2022-10-04 12:40:33,974 INFO mapreduce.Job: Running job: job_1664901602347_0001
2022-10-04 12:40:45,146 INFO mapreduce.Job: Job job_1664901602347_0001 running in uber mode : false
2022-10-04 12:40:45,147 INFO mapreduce.Job:  map 0% reduce 0%
2022-10-04 12:40:52,319 INFO mapreduce.Job:  map 100% reduce 0%
2022-10-04 12:40:59,512 INFO mapreduce.Job:  map 100% reduce 100%
2022-10-04 12:41:00,531 INFO mapreduce.Job: Job job_1664901602347_0001 completed successfully
2022-10-04 12:41:00,694 INFO mapreduce.Job: Counters: 54
        File System Counters
                FILE: Number of bytes read=98
                FILE: Number of bytes written=552153
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=191
                HDFS: Number of bytes written=60
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
                HDFS: Number of bytes read erasure-coded=0
        Job Counters 
                Launched map tasks=1
                Launched reduce tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=4860
                Total time spent by all reduces in occupied slots (ms)=5021
                Total time spent by all map tasks (ms)=4860
                Total time spent by all reduce tasks (ms)=5021
                Total vcore-milliseconds taken by all map tasks=4860
                Total vcore-milliseconds taken by all reduce tasks=5021
                Total megabyte-milliseconds taken by all map tasks=4976640
                Total megabyte-milliseconds taken by all reduce tasks=5141504
        Map-Reduce Framework
                Map input records=6
                Map output records=16
                Map output bytes=154
                Map output materialized bytes=98
                Input split bytes=100
                Combine input records=16
                Combine output records=8
                Reduce input groups=8
                Reduce shuffle bytes=98
                Reduce input records=8
                Reduce output records=8
                Spilled Records=16
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=235
                CPU time spent (ms)=1180
                Physical memory (bytes) snapshot=318394368
                Virtual memory (bytes) snapshot=5478535168
                Total committed heap usage (bytes)=141692928
                Peak Map Physical memory (bytes)=211103744
                Peak Map Virtual memory (bytes)=2734690304
                Peak Reduce Physical memory (bytes)=107290624
                Peak Reduce Virtual memory (bytes)=2743844864
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters 
                Bytes Read=91
        File Output Format Counters 
                Bytes Written=60