MapReduce 作业（用 python 编写）在 EMR 上运行缓慢

Posted 2023-04-18

【中文标题】MapReduce 作业（用 python 编写）在 EMR 上运行缓慢

【英文标题】：MapReduce Job (written in python) run slow on EMR

【发布时间】：2015-02-22 15:53:56

【问题描述】：

我正在尝试使用 python 的 MRJob 包编写 MapReduce 作业。该作业处理存储在 S3 中的约 36,000 个文件。每个文件约为 2MB。当我在本地运行作业（将 S3 存储桶下载到我的计算机）时，运行大约需要 1 小时。但是，当我尝试在 EMR 上运行它时，它需要更长的时间（我在 8 小时时停止了它，它在映射器中完成了 10%）。我在下面附上了我的 mapper_init 和 mapper 的代码。有谁知道什么会导致这样的问题？有谁知道如何修理它？我还应该注意，当我将输入限制为 100 个文件的样本时，它可以正常工作。

def mapper_init(self):
    """
    Set class variables that will be useful to our mapper:
        filename: the path and filename to the current recipe file
        previous_line: The line previously parsed. We need this because the
          ingredient name is in the line after the tag
    """

    #self.filename = os.environ["map_input_file"]  # Not currently used
    self.previous_line = "None yet"
    # Determining if an item is in a list is O(n) while determining if an
    #  item is in a set is O(1)
    self.stopwords = set(stopwords.words('english'))
    self.stopwords = set(self.stopwords_list)


def mapper(self, _, line):
    """
    Takes a line from an html file and yields ingredient words from it

    Given a line of input from an html file, we check to see if it
    contains the identifier that it is an ingredient. Due to the
    formatting of our html files from allrecipes.com, the ingredient name
    is actually found on the following line. Therefore, we save the
    current line so that it can be referenced in the next pass of the
    function to determine if we are on an ingredient line.

    :param line: a line of text from the html file as a str
    :yield: a tuple containing each word in the ingredient as well as a
        counter for each word. The counter is not currently being used,
        but is left in for future development. e.g. "chicken breast" would
        yield "chicken" and "breast"
    """

    # TODO is there a better way to get the tag?
    if re.search(r'span class="ingredient-name" id="lblIngName"',
                 self.previous_line):
        self.previous_line = line
        line = self.process_text(line)
        line_list = set(line.split())
        for word in line_list:
            if word not in self.stopwords:
                yield (word, 1)
    else:
        self.previous_line = line
    yield ('', 0)

【参考方案1】：

问题是您有更多的小文件。添加使用 s3distcp 将文件复制到 EMR 的引导步骤。在使用 s3distcp 时尝试将小文件聚合到 ~128MB 文件中。

Hadoop 不适合处理大量小文件。

由于您是手动将文件下载到计算机并运行，因此运行速度更快。

使用 S3distCP 将文件复制到 EMR 后，使用 HDFS 中的文件。

【讨论】：

如何在亚马逊 emr 上使用 hdfs 而不是 s3？