hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <omal...@apache.org>
Subject Re: How are records with equal key sorted in hadoop-0.18?
Date Mon, 08 Dec 2008 17:10:05 GMT
On Dec 8, 2008, at 8:02 AM, Christian Kunz wrote:

> Comparing hadoop-default.xml of hadoop-0.18 with hadoop-0.17, didn't
> map.sort.class change from
> org.apache.hadoop.mapred.MergeSorter to
> org.apache.hadoop.util.QuickSort?

Yes, but the quick sort is only used in the mapper. The reducer  
already has sorted runs and therefore only needs an external merge sort.

The primary change in 0.18 for the reducer was HADOOP-2095. What were  
the values of io.sort.factor and io.file.buffer.size?

Christian, can you get the heap profile for one of the reduces that is  
failing?

mapred.task.profile=true
mapred.task.profile.maps= <empty string, since we don't want any maps>
mapred.task.profile.reduces= <number of reduce to profile>
mapred.task.profile.params=- 
agentlib:hprof=heap=sites,force=n,thread=y,verbose=n,file=%s

-- Owen

Mime
View raw message