CDH- 测试mr

Posted 2020-11-19 rzcong

cdh的mr样例算法的jar包在

[[email protected]172-32-1-221 hadoop-0.20-mapreduce]$ pwd
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop-0.20-mapreduce

查看该目录下的文件

[[email protected]172-32-1-221 hadoop-0.20-mapreduce]$ ll
total 400
drwxr-xr-x 2 root root   4096 Jan 20  2017 bin
-rw-r--r-- 1 root root 348776 Jan 20  2017 CHANGES.txt
drwxr-xr-x 2 root root   4096 Jan 20  2017 cloudera
lrwxrwxrwx 1 root root     16 Jul 20  2017 conf -> /etc/hadoop/conf
drwxr-xr-x 6 root root   4096 Jan 20  2017 contrib
drwxr-xr-x 4 root root   4096 Jan 20  2017 example-confs
lrwxrwxrwx 1 root root     45 Jul 20  2017 hadoop-ant-2.6.0-mr1-cdh5.10.0.jar -> ../../jars/hadoop-ant-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     34 Jul 20  2017 hadoop-ant-mr1.jar -> hadoop-ant-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     46 Jul 20  2017 hadoop-core-2.6.0-mr1-cdh5.10.0.jar -> ../../jars/hadoop-core-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     35 Jul 20  2017 hadoop-core-mr1.jar -> hadoop-core-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     50 Jul 20  2017 hadoop-examples-2.6.0-mr1-cdh5.10.0.jar -> ../../jars/hadoop-examples-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     23 Jul 20  2017 hadoop-examples.jar -> hadoop-examples-mr1.jar
lrwxrwxrwx 1 root root     39 Jul 20  2017 hadoop-examples-mr1.jar -> hadoop-examples-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     46 Jul 20  2017 hadoop-test-2.6.0-mr1-cdh5.10.0.jar -> ../../jars/hadoop-test-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     35 Jul 20  2017 hadoop-test-mr1.jar -> hadoop-test-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     47 Jul 20  2017 hadoop-tools-2.6.0-mr1-cdh5.10.0.jar -> ../../jars/hadoop-tools-2.6.0-mr1-cdh5.10.0.jar
lrwxrwxrwx 1 root root     36 Jul 20  2017 hadoop-tools-mr1.jar -> hadoop-tools-2.6.0-mr1-cdh5.10.0.jar
drwxr-xr-x 3 root root   4096 Jan 20  2017 include
drwxr-xr-x 5 root root   4096 Jan 20  2017 lib
-rw-r--r-- 1 root root  13366 Jan 20  2017 LICENSE.txt
-rw-r--r-- 1 root root    101 Jan 20  2017 NOTICE.txt
-rw-r--r-- 1 root root   1366 Jan 20  2017 README.txt
drwxr-xr-x 3 root root   4096 Jan 20  2017 sbin
drwxr-xr-x 5 root root   4096 Jan 20  2017 webapps

可以用hadoop-examples.jar里面的wordcount做测试

#hadoop jar hadoop-examples.jar

可以看到里面都有些上面可以使用的类

An example program must be given as the first argument.
 Valid program names are:
 aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
 aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
 bbp: A map/reduce program that uses Bailey-Borwein-Plouffe to compute exact digits of Pi.
 dbcount: An example job that count the pageview counts from a database.
 distbbp: A map/reduce program that uses a BBP-type formula to compute exact bits of Pi.
 grep: A map/reduce program that counts the matches of a regex in the input.
 join: A job that effects a join over sorted, equally partitioned datasets
 multifilewc: A job that counts words from several files.
 pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
 pi: A map/reduce program that estimates Pi using a quasi-Monte Carlo method.
 randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
 randomwriter: A map/reduce program that writes 10GB of random data per node.
 secondarysort: An example defining a secondary sort to the reduce.
 sort: A map/reduce program that sorts the data written by the random writer.
 sudoku: A sudoku solver.
 teragen: Generate data for the terasort
 terasort: Run the terasort
 teravalidate: Checking results of terasort
 wordcount: A map/reduce program that counts the words in the input files.
 wordmean: A map/reduce program that counts the average length of the words in the input files.
 wordmedian: A map/reduce program that counts the median length of the words in the input files.
 wordstandarddeviation: A map/reduce program that counts the standard deviation of the length of the words in the input files.

这里我直接取wordcount类来做测试，首先上传文件到hdfs准备好计算

hdfs dfs -mkdir /user/zc.lee/input/
hdfs dfs -put /user/PG/conf/type.txt /user/zc.lee/input/

开始计算

hadoop jar hadoop-examples.jar wordcount /user/zc.lee/input/type.txt /user/zc.lee/ouputtest

检查结果

hdfs dfs -text /user/zc.lee/ouputtest/*