hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron_ <r...@sfu.ca>
Subject Hadoop Reduce is never called
Date Sat, 30 Oct 2010 19:47:04 GMT

Hi,

I am new to Hadoop and new to the community.  I am using Hadoop 0.20.2 and
working through the LineIndexer example.  I  have defined the Reducer using
the following line:

job.setReducerClass (LineIndexerReducer.class);

The full code is in  http://old.nabble.com/file/p30094575/LineIndexer.java
LineIndexer.java .  My Reducer code is 
http://old.nabble.com/file/p30094575/LineIndexerReducer.java
LineIndexerReducer.java .  I have a trace print statement in the reduce
method of the Reducer c;ass.  This println statement never gets executed
indicating that my Reducer never gets called.  My mapper, 
http://old.nabble.com/file/p30094575/LineIndexerMapper.java
LineIndexerMapper.java  works correctly.

Does anyone have any idea what I have done wrong?

Here is the result:
10/10/30 11:54:08 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
core-default.xml, mapred-default.xml and hdfs-default.xml respectively
10/10/30 11:54:08 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
10/10/30 11:54:08 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/10/30 11:54:08 INFO input.FileInputFormat: Total input paths to process :
1
10/10/30 11:54:08 INFO mapred.JobClient: Running job: job_local_0001
10/10/30 11:54:08 INFO input.FileInputFormat: Total input paths to process :
1
10/10/30 11:54:09 INFO mapred.MapTask: io.sort.mb = 100
10/10/30 11:54:09 INFO mapred.MapTask: data buffer = 79691776/99614720
10/10/30 11:54:09 INFO mapred.MapTask: record buffer = 262144/327680
10/10/30 11:54:09 INFO mapred.MapTask: Starting flush of map output
10/10/30 11:54:09 INFO mapred.JobClient:  map 0% reduce 0%
10/10/30 11:54:10 INFO mapred.MapTask: Finished spill 0
10/10/30 11:54:10 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0
is done. And is in the process of commiting
10/10/30 11:54:10 INFO mapred.LocalJobRunner: 
10/10/30 11:54:10 INFO mapred.TaskRunner: Task
'attempt_local_0001_m_000000_0' done.
10/10/30 11:54:10 INFO mapred.LocalJobRunner: 
10/10/30 11:54:10 INFO mapred.Merger: Merging 1 sorted segments
10/10/30 11:54:10 INFO mapred.Merger: Down to the last merge-pass, with 1
segments left of total size: 1904864 bytes
10/10/30 11:54:10 INFO mapred.LocalJobRunner: 
10/10/30 11:54:10 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0
is done. And is in the process of commiting
10/10/30 11:54:10 INFO mapred.LocalJobRunner: 
10/10/30 11:54:10 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0
is allowed to commit now
10/10/30 11:54:10 INFO output.FileOutputCommitter: Saved output of task
'attempt_local_0001_r_000000_0' to output_331
10/10/30 11:54:10 INFO mapred.LocalJobRunner: reduce > reduce
10/10/30 11:54:10 INFO mapred.TaskRunner: Task
'attempt_local_0001_r_000000_0' done.
10/10/30 11:54:10 INFO mapred.JobClient:  map 100% reduce 100%
10/10/30 11:54:10 INFO mapred.JobClient: Job complete: job_local_0001
10/10/30 11:54:10 INFO mapred.JobClient: Counters: 12
10/10/30 11:54:10 INFO mapred.JobClient:   FileSystemCounters
10/10/30 11:54:10 INFO mapred.JobClient:     FILE_BYTES_READ=2318398
10/10/30 11:54:10 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=5749266
10/10/30 11:54:10 INFO mapred.JobClient:   Map-Reduce Framework
10/10/30 11:54:10 INFO mapred.JobClient:     Reduce input groups=7275
10/10/30 11:54:10 INFO mapred.JobClient:     Combine output records=0
10/10/30 11:54:10 INFO mapred.JobClient:     Map input records=6045
10/10/30 11:54:10 INFO mapred.JobClient:     Reduce shuffle bytes=0
10/10/30 11:54:10 INFO mapred.JobClient:     Reduce output records=32242
10/10/30 11:54:10 INFO mapred.JobClient:     Spilled Records=64484
10/10/30 11:54:10 INFO mapred.JobClient:     Map output bytes=1840378
10/10/30 11:54:10 INFO mapred.JobClient:     Combine input records=0
10/10/30 11:54:10 INFO mapred.JobClient:     Map output records=32242
10/10/30 11:54:10 INFO mapred.JobClient:     Reduce input records=32242

Thank you in advance,

Cameron



-- 
View this message in context: http://old.nabble.com/Hadoop-Reduce-is-never-called-tp30094575p30094575.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Mime
View raw message