hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From maha <m...@umail.ucsb.edu>
Subject Bug in my configurations, help!
Date Sun, 27 Feb 2011 04:00:19 GMT
Hi,

Even though I'm running a very simple job  of 4 small records (2 per map) and one reducer
.. I still get all 9 records output of map spilled. 

But map-logs shows spilled to be ZERO !
2011-02-26 16:05:35,301 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1
2011-02-26 16:05:35,307 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
2011-02-26 16:05:35,467 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
2011-02-26 16:05:35,467 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
2011-02-26 16:05:35,491 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
2011-02-26 16:05:35,571 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0   <-----------------------
WHY IS THIS ZERO WHEN FINAL JOB COUNTER SAYS IT'S 9 SPILLED RECORDS FROM MAPS!
I'm using default configurations, I tried changing the factor and the sort.mb parameters but
no way.  Is that how it's supposed to be ???

Please any idea would be helpful. 

Thank you,
Maha
Mime
View raw message