hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-875) memory model is not accurate enough for map side sorts
Date Thu, 11 Jan 2007 15:14:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463912

Devaraj Das commented on HADOOP-875:

Could you please let me know whether there were multiple spills-to-disk of map outputs for
those failed maps?

> memory model is not accurate enough for map side sorts
> ------------------------------------------------------
>                 Key: HADOOP-875
>                 URL: https://issues.apache.org/jira/browse/HADOOP-875
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.10.0
>            Reporter: Owen O'Malley
>         Assigned To: Devaraj Das
> I configured a sort (IdentityMapper) with large compressed values with the map output
buffer (io.sort.mb) set to 300mb and the child jvm heap size set to 900mb and some of the
maps were running out of memory deterministically. I suspect that some part of the data path
is not handling large values well and is consuming large amounts of ram.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message