如何在 hive 中设置 mapreduce 任务的数量等于 1

Posted 2023-04-17

【中文标题】如何在 hive 中设置 mapreduce 任务的数量等于 1

【英文标题】：How to set number of mapreduce task equal to 1 in hive

【发布时间】：2011-12-27 10:42:17

【问题描述】：

我尝试在蜂巢中关注-

set hive.exec.reducers.max = 1;
set mapred.reduce.tasks = 1;

from flat_json
insert overwrite table aggr_pgm_measure PARTITION(dt='$START_TIME')
reduce  log_time,
 req_id, ac_id, client_key, rulename, categoryname, bsid, visitorid, visitorgroupid, visitortargetid, targetpopulationid, windowsessionid, eventseq, event_code, eventstarttime
 using '$SCRIPT_LOC/aggregator.pl' as 
 metric_id, metric_value, aggr_type, rule_name, category_name; 

尽管将最大数量和减少任务的数量设置为 1，但我看到生成了 2 个映射减少任务。请看下面-

 > set hive.exec.reducers.max = 1;
hive>  set mapred.reduce.tasks = 1;
hive>
    > from flat_json
    > insert overwrite table aggr_pgm_measure PARTITION(dt='$START_TIME')
    > reduce  log_time,
    >  req_id, ac_id, client_key, rulename, categoryname, bsid, visitorid, visitorgroupid, visitortargetid, targetpopulationid, windowsessionid, eventseq, event_code, eventstarttime
    >  using '$SCRIPT_LOC/aggregator.pl' as
    >  metric_id, metric_value, aggr_type, rule_name, category_name;
converting to local s3://dsp-emr-test/anurag/dsp-test/60mins/script/aggregator.pl
Added resource: /mnt/var/lib/hive_07_1/downloaded_resources/aggregator.pl
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201112270825_0009, Tracking URL = http://ip-10-85-66-9.ec2.internal:9100/jobdetails.jsp?jobid=job_201112270825_0009
Kill Command = /home/hadoop/.versions/0.20.205/libexec/../bin/hadoop job  -Dmapred.job.tracker=10.85.66.9:9001 -kill job_201112270825_0009
2011-12-27 10:30:03,542 Stage-1 map = 0%,  reduce = 0%

【参考方案1】：

你认为相关的两件事并不相关。您正在设置减少 任务 的数量，而不是 MapReduce 作业。 Hive 会将您的查询转换为几个 MapReduce 作业，就像需要完成的性质一样。每个 MapReduce 作业由多个 map tasks 和 reduce tasks 组成。

您设置的是任务的最大数量。这意味着，每个 MapReduce 作业都将受到它可以启动的任务数量的限制。不过，您仍然需要运行两个作业。对于 Hive 的 MapReduce 作业数量，您无能为力。它需要运行每个阶段才能执行您的查询。