hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Performance issues with large map/reduce processes
Date Thu, 27 Dec 2007 20:03:53 GMT

That is a very small heap.  The reduces, in particular, would benefit
sustantially from having more memory.

Other than that (and having fewer reduces), I am at a bit of a loss.  I know
that others are working on comparably sized problems without much
difficulty.  

There might be an interaction with your small heaps and attempts to buffer
outputs.


On 12/27/07 10:31 AM, "jag123" <jaganrvce@yahoo.com> wrote:

>> What happens on the large size problems if you decrease the number of
>> maps,
>> but keep input size constant?
> 
> My map processes run out of Heap space if I decrease the number of map
> processes. 
> The maximum number of records that my map process can handle is 250k as my
> heap size is only 256m.


Mime
View raw message