hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2959) When a mapper needs to run a combiner, it should create one and reuse it, instead of creating one per partition per spill
Date Fri, 07 Mar 2008 03:58:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576029#action_12576029
] 

Runping Qi commented on HADOOP-2959:
------------------------------------


I should have pounted out the use case why this is matter.
When the combining logic (reducer logic) depends on some thing that is initialized 
in the configure method, and of the configure method call is relative expensive (say initialize
a dictionary 
from a file on dfs), then such an optimization makes a huge difference.



> When a mapper needs to run a combiner, it should create one and reuse it, instead of
creating one per partition per spill
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-2959
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2959
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.16.0
>            Reporter: Runping Qi
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message