hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Kubes <nutch-...@dragonflymc.com>
Subject Re: Out of Memory during Sorts
Date Mon, 12 Jun 2006 16:22:06 GMT
Unfortunately I don't have the stack traces anymore.  The java heap size 
was set to 2G.  When I upped the sort factor I get alot more IO, but it 
finishes and doesn't run out of memory.

Dennis

Owen O'Malley wrote:
>
> On Jun 11, 2006, at 8:07 AM, Dennis Kubes wrote:
>
>> Can someone lead me in the right direction as to configuring settings 
>> for large sorting operations > 1M rows.  I keep getting out of memory 
>> exceptions during the sort phase.  Here are my current settings.  I 
>> have 2G heap space on each box.
>
> In general, it would help to have stack traces so that we could tell 
> what part of the sort was running out of memory. What was your jvm 
> heap size set to? It is interesting that increasing the sort factor 
> got you around the problem.
>
> -- Owen
>

Mime
View raw message