hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3104) MultithreadMapRunner keeps consuming records even if trheads are not available
Date Fri, 28 Mar 2008 05:52:24 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582920#action_12582920
] 

Amar Kamat commented on HADOOP-3104:
------------------------------------

Here are some comments
1) The javadoc comments should not mention the default value. That might change and will require
code change too. So you can keep the earlier comment as is and just add the comment about
the wait parameter.
2) I think mapred.map.multithreadedrunner.backoff seems more appropriate than mapred.map.multithreadedrunner.waitwhennothreads,
comments?
3) 10ms seems too short. I was wondering what if we double it everytime. Something like 10,20,40,80
...  


> MultithreadMapRunner keeps consuming records even if trheads are not available
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-3104
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3104
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.16.1
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Critical
>             Fix For: 0.16.2
>
>         Attachments: patch3104.txt
>
>
> The ExecutorService execute() method does not block when there are not threads available,
it queues up the runnables until there are threads. 
> The problem is that all key/values are read and kept in memory for the task, with large
datasets this will create a OOM exception.
> Have to figure out how to use the execute in blocking fashion.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message