从未为大数据创建的 Hadoop 减少输出文件

Posted 2023-03-06

【中文标题】从未为大数据创建的 Hadoop 减少输出文件

【英文标题】：Hadoop Reduce Output File Never Created for Large Data

【发布时间】：2013-05-14 19:51:31

【问题描述】：

我正在 Hadoop 1.1.1 (Ubuntu) 上用 Java 编写一个应用程序，它比较字符串以找到最长的公共子字符串。我已经为小型数据集成功运行了 map 和 reduce 阶段。每当我增加输入的大小时，我的减少输出永远不会出现在我的目标输出目录中。它一点也不抱怨，这让这一切变得更加奇怪。我在 Eclipse 中运行所有东西，我有 1 个映射器和 1 个减速器。

我的 reducer 在字符串集合中找到最长的公共子字符串，然后将子字符串作为键发出，并将包含它的字符串的索引作为值发出。我有一个简短的例子。

输入数据

0: ALPHAA

1: ALPHAB

2: ALZHA

输出输出

Key: ALPHA  Value: 0

Key: ALPHA  Value: 1

Key: AL  Value: 0

Key: AL  Value: 1

Key: AL  Value: 2

前两个输入字符串都共享“ALPHA”作为公共子字符串，而所有三个都共享“AL”。当过程完成时，我最终索引子字符串并将它们写入数据库。

另外观察，我可以看到在我的输出目录中创建了中间文件，只是减少的数据从未放入输出文件中。

我在下面粘贴了 Hadoop 输出日志，它声称它有许多来自 reducer 的输出记录，只是它们似乎消失了。任何建议表示赞赏。

Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Use GenericOptionsParser for parsing the arguments. Applications should implement Tool     for the same.
No job jar file set.  User classes may not be found. See JobConf(Class) or     JobConf#setJar(String).
Total input paths to process : 1
Running job: job_local_0001
setsid exited with exit code 0
 Using ResourceCalculatorPlugin :     org.apache.hadoop.util.LinuxResourceCalculatorPlugin@411fd5a3
Snappy native library not loaded
io.sort.mb = 100
data buffer = 79691776/99614720
record buffer = 262144/327680
 map 0% reduce 0%
Spilling map output: record full = true
bufstart = 0; bufend = 22852573; bufvoid = 99614720
kvstart = 0; kvend = 262144; length = 327680
Finished spill 0
Starting flush of map output
Finished spill 1
Merging 2 sorted segments
Down to the last merge-pass, with 2 segments left of total size: 28981648 bytes

Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting

Task attempt_local_0001_m_000000_0 done.
 Using ResourceCalculatorPlugin :     org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3aff2f16

Merging 1 sorted segments
Down to the last merge-pass, with 1 segments left of total size: 28981646 bytes

 map 100% reduce 0%
reduce > reduce
 map 100% reduce 66%
reduce > reduce
 map 100% reduce 67%
reduce > reduce
reduce > reduce
 map 100% reduce 68%
reduce > reduce
reduce > reduce
reduce > reduce
 map 100% reduce 69%
reduce > reduce
reduce > reduce
 map 100% reduce 70%
reduce > reduce
job_local_0001
Job complete: job_local_0001
Counters: 22
  File Output Format Counters 
    Bytes Written=14754916
  FileSystemCounters
    FILE_BYTES_READ=61475617
    HDFS_BYTES_READ=97361881
    FILE_BYTES_WRITTEN=116018418
    HDFS_BYTES_WRITTEN=116746326
  File Input Format Counters 
    Bytes Read=46366176
  Map-Reduce Framework
    Reduce input groups=27774
    Map output materialized bytes=28981650
    Combine output records=0
    Map input records=4629524
    Reduce shuffle bytes=0
    Physical memory (bytes) snapshot=0
    Reduce output records=832559
    Spilled Records=651304
    Map output bytes=28289481
    CPU time spent (ms)=0
    Total committed heap usage (bytes)=2578972672
    Virtual memory (bytes) snapshot=0
    Combine input records=0
    Map output records=325652
    SPLIT_RAW_BYTES=136
    Reduce input records=27774
reduce > reduce
reduce > reduce

【问题讨论】：

想一想，也许您的代码中有错误，大型运行中的某些字符串会暴露？

我认为你是对的。我刚刚发现了一个异常。

【参考方案1】：

我将reduce() 和map() 逻辑放在try-catch 块中，catch 块递增一个计数器，其组为“异常”，其名称为异常消息。这给了我一个快速的方法（通过查看计数器列表）来查看抛出了哪些异常（如果有）。