Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <27953431.1168528467514.JavaMail.jira@brutus>
Date: Thu, 11 Jan 2007 07:14:27 -0800 (PST)
From: "Devaraj Das (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-875) memory model is not accurate enough
 for map side sorts
In-Reply-To: <18649543.1168447167858.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463912 ] 

Devaraj Das commented on HADOOP-875:
------------------------------------

Could you please let me know whether there were multiple spills-to-disk of map outputs for those failed maps?

> memory model is not accurate enough for map side sorts
> ------------------------------------------------------
>
>                 Key: HADOOP-875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-875
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.10.0
>            Reporter: Owen O'Malley
>         Assigned To: Devaraj Das
>
> I configured a sort (IdentityMapper) with large compressed values with the map output buffer (io.sort.mb) set to 300mb and the child jvm heap size set to 900mb and some of the maps were running out of memory deterministically. I suspect that some part of the data path is not handling large values well and is consuming large amounts of ram.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira