在 CDH4 集群上运行 Pig 时无法进入 mapreduce 模式（Hadoop 2 + MapReduce v1）

Posted 2023-03-24

技术标签:

【中文标题】在 CDH4 集群上运行 Pig 时无法进入 mapreduce 模式（Hadoop 2 + MapReduce v1）【英文标题】：Cannot get into mapreduce mode while running Pig on CDH4 cluster (Hadoop 2 + MapReduce v1) 【发布时间】：2013-02-20 19:09:13 【问题描述】：

我想在包含 3 个 Amazon 实例的 CDH4 集群上运行 pig 脚本（非嵌入式和嵌入式）。我为位于 /home/ubuntu/core-site.xml 的 pig 创建了一个伪造的配置文件（但地址是正确的），如下所示：

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://server1:8020</value>
  </property>
  <property>
    <name>mapred.job.tracker</name>
    <value>server1:8021</value>
  </property>
</configuration>

当我尝试运行时：

ubuntu@server1:~$ export HADOOP_CONF_DIR=/home/ubuntu
ubuntu@server1:~$ export PIG_CLASSPATH=/home/ubuntu
ubuntu@server1:~$ pig -x mapreduce -f test.pig

脚本运行并返回了正确的结果，但在控制台中我发现了很多“LocalJobRunner”，并且在 MapReduce 作业跟踪器 Web 界面中没有报告作业。谁能告诉我为什么它不在 mapreduce 模式下运行，为什么它不报告这种情况的任何错误？如何在 mapreduce 模式下运行它？

我的集群是 4.1.3（新安装的，所有配置都是默认的）， Pig 0.10.0-cdh4.1.3。

2013-02-20 18:57:15,843 [main] INFO  org.apache.pig.Main - Apache Pig version 0.10.0-cdh4.1.3 (rexported) compiled Jan 26 2013, 17:35:45
2013-02-20 18:57:15,843 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/ubuntu/epi-tre/pig_1361386635821.log
2013-02-20 18:57:16,519 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://server1:8020
2013-02-20 18:57:16,524 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:17,249 [main] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:17,826 [main] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-02-20 18:57:17,827 [main] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
2013-02-20 18:57:17,828 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-02-20 18:57:17,829 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
2013-02-20 18:57:17,830 [main] WARN  org.apache.hadoop.conf.Configuration - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-02-20 18:57:18,102 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
2013-02-20 18:57:18,460 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-02-20 18:57:18,475 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-02-20 18:57:18,507 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2013-02-20 18:57:18,508 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2013-02-20 18:57:18,530 [main] WARN  org.apache.hadoop.conf.Configuration - session.id is deprecated. Instead, use dfs.metrics.session-id
2013-02-20 18:57:18,531 [main] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2013-02-20 18:57:18,553 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-02-20 18:57:18,563 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-02-20 18:57:18,566 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2458140745309154844.jar
2013-02-20 18:57:22,423 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2458140745309154844.jar created
2013-02-20 18:57:22,453 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-02-20 18:57:22,544 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-02-20 18:57:22,553 [Thread-4] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2013-02-20 18:57:22,581 [Thread-4] WARN  org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-02-20 18:57:22,735 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:22,735 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:22,790 [Thread-4] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-02-20 18:57:22,790 [Thread-4] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-02-20 18:57:22,851 [Thread-4] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-02-20 18:57:23,045 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-02-20 18:57:23,055 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:23,057 [Thread-4] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:23,141 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter set in config null
2013-02-20 18:57:23,178 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.df.interval is deprecated. Instead, use fs.df.interval
2013-02-20 18:57:23,178 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.max.objects is deprecated. Instead, use dfs.namenode.max.objects
2013-02-20 18:57:23,187 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2013-02-20 18:57:23,187 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.access.time.precision is deprecated. Instead, use dfs.namenode.accesstime.precision
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir
2013-02-20 18:57:23,188 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.considerLoad is deprecated. Instead, use dfs.namenode.replication.considerLoad
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.balance.bandwidthPerSec is deprecated. Instead, use dfs.datanode.balance.bandwidthPerSec
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.threshold.pct is deprecated. Instead, use dfs.namenode.safemode.threshold-pct
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.http.address is deprecated. Instead, use dfs.namenode.http-address
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.name.dir.restore is deprecated. Instead, use dfs.namenode.name.dir.restore
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.https.client.keystore.resource is deprecated. Instead, use dfs.client.https.keystore.resource
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.address is deprecated. Instead, use dfs.namenode.backup.address
2013-02-20 18:57:23,189 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.backup.http.address is deprecated. Instead, use dfs.namenode.backup.http-address
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions is deprecated. Instead, use dfs.permissions.enabled
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.safemode.extension is deprecated. Instead, use dfs.namenode.safemode.extension
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.datanode.max.xcievers is deprecated. Instead, use dfs.datanode.max.transfer.threads
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.https.need.client.auth is deprecated. Instead, use dfs.client.https.need-auth
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.https.address is deprecated. Instead, use dfs.namenode.https-address
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.replication.interval is deprecated. Instead, use dfs.namenode.replication.interval
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.write.packet.size is deprecated. Instead, use dfs.client-write-packet-size
2013-02-20 18:57:23,190 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.permissions.supergroup is deprecated. Instead, use dfs.permissions.superusergroup
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - topology.script.number.args is deprecated. Instead, use net.topology.script.number.args
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - dfs.secondary.http.address is deprecated. Instead, use dfs.namenode.secondary.http-address
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.checkpoint.period is deprecated. Instead, use dfs.namenode.checkpoint.period
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - topology.node.switch.mapping.impl is deprecated. Instead, use net.topology.node.switch.mapping.impl
2013-02-20 18:57:23,191 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:23,193 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter
2013-02-20 18:57:23,246 [Thread-5] WARN  mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-20 18:57:23,333 [Thread-5] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2013-02-20 18:57:23,344 [Thread-5] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3bfc47
2013-02-20 18:57:23,362 [Thread-5] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed hdfs://server1:8020/tmp/cf_rating.10000.csv:0+676168
2013-02-20 18:57:23,374 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
2013-02-20 18:57:23,507 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - data buffer = 79691776/99614720
2013-02-20 18:57:23,507 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - record buffer = 262144/327680
2013-02-20 18:57:23,639 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2013-02-20 18:57:24,485 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:24,488 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - Starting flush of map output
2013-02-20 18:57:25,059 [Thread-5] INFO  org.apache.hadoop.mapred.MapTask - Finished spill 0
2013-02-20 18:57:25,065 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
2013-02-20 18:57:25,087 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,098 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_m_000000_0' done.
2013-02-20 18:57:25,104 [Thread-5] WARN  mapreduce.Counters - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2013-02-20 18:57:25,130 [Thread-5] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@461d318f
2013-02-20 18:57:25,130 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,137 [Thread-5] INFO  org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2013-02-20 18:57:25,164 [Thread-5] INFO  org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 25 bytes
2013-02-20 18:57:25,164 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,336 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
2013-02-20 18:57:25,337 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-02-20 18:57:25,338 [Thread-5] WARN  org.apache.hadoop.conf.Configuration - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2013-02-20 18:57:25,339 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - 
2013-02-20 18:57:25,341 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task attempt_local_0001_r_000000_0 is allowed to commit now
2013-02-20 18:57:25,371 [Thread-5] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://server1:8020/tmp/temp-1313079537/tmp188373172
2013-02-20 18:57:25,371 [Thread-5] INFO  org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
2013-02-20 18:57:25,373 [Thread-5] INFO  org.apache.hadoop.mapred.Task - Task 'attempt_local_0001_r_000000_0' done.
2013-02-20 18:57:28,144 [main] WARN  org.apache.pig.tools.pigstats.PigStatsUtil - Failed to get RunningJob for job job_local_0001
2013-02-20 18:57:28,147 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-02-20 18:57:28,151 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
2.0.0-cdh4.1.3  0.10.0-cdh4.1.3 ubuntu  2013-02-20 18:57:18 2013-02-20 18:57:28 GROUP_BY

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime  MinMapTIme  AvgMapTime  MaxReduceTime   MinReduceTimeAvgReduceTime  Alias   Feature Outputs
job_local_0001  1   1   n/a n/a n/a n/a n/a n/a a,b,c   GROUP_BY,COMBINER   hdfs://server1:8020/tmp/temp-1313079537/tmp188373172,

Input(s):
Successfully read 0 records from: "/tmp/cf_rating.10000.csv"

Output(s):
Successfully stored 0 records in: "hdfs://server1:8020/tmp/temp-1313079537/tmp188373172"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local_0001


2013-02-20 18:57:28,151 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2013-02-20 18:57:28,158 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-02-20 18:57:28,159 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(10000)

【问题讨论】：

【参考方案1】：

原来是我忘记部署 MapReduce“客户端配置”，所以我的整个集群都在本地模式下运行：">已解决。

【讨论】：

愿意分享您所做的更改吗？抱歉一段时间过去了，我不知道我做了什么。也许我在配置中更改了一些端口，但忘记将其部署到客户端配置。

以上是关于在 CDH4 集群上运行 Pig 时无法进入 mapreduce 模式（Hadoop 2 + MapReduce v1）的主要内容，如果未能解决你的问题，请参考以下文章

Pig local vs mapreduce 模型性能对比

无法在 Amazon EMR 集群上使用 PIG 0.12.0 和 Hadoop 2.4.0 找到 MySql 驱动程序

hadoop hdfs 从 cdh3 迁移到 cdh4

在 Amazon AWS 上并行运行 Pig 脚本

CDH4 将 Cloudera Manager 还原到现有集群

Pig 无法处理大文件