hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <o...@yahoo-inc.com>
Subject Re: Out of Memory during Sorts
Date Tue, 13 Jun 2006 04:50:16 GMT

On Jun 11, 2006, at 8:07 AM, Dennis Kubes wrote:

> Can someone lead me in the right direction as to configuring settings 
> for large sorting operations > 1M rows.  I keep getting out of memory 
> exceptions during the sort phase.  Here are my current settings.  I 
> have 2G heap space on each box.

Please check the value of "mapred.child.java.opts". It controls the 
options for heap sized allocated to the Task.  It defaults to 200m, 
which seems really low given that the default io.sort.mb is 100m (and 
yours is set to 200m). Try increasing the max heap size to 1024m or so.

-- Owen

View raw message