hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6144) DefaultSpeculator always add both MAP and REDUCE Speculative task even MAP_SPECULATIVE or REDUCE_SPECULATIVE is disabled.
Date Mon, 24 Nov 2014 23:16:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223738#comment-14223738
] 

Xuan Gong commented on MAPREDUCE-6144:
--------------------------------------

Thanks for working on this. [~zxu]
One question here: do we really need this ? I do not think we need to do this. Please correct
me if I am wrong.
We have SpeculatorEventDispatcher in MRAppMaster which checks the configurations: MRJobConfig.MAP_SPECULATIVE
and MRJobConfig.REDUCE_SPECULATIVE, If any of the configurations are false, the Map or reduce
task will not be handled by speculator. 
And every time when we launched a container, we will send SpeculatorEvent, too. If the related
configuration is set as false, this even will not be handled. In that case, it will not handle
any SpeculatorEvents. We need to handle SpeculatorEvents to keep the status update. If we
do not handle any such events, when we tries to call computeSpeculations, it will not schedule
any speculations.




> DefaultSpeculator always add both MAP and REDUCE Speculative task even MAP_SPECULATIVE
or REDUCE_SPECULATIVE is disabled.
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6144
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6144
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 2.5.1
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>         Attachments: MAPREDUCE-6144.000.patch, MAPREDUCE-6144.001.patch
>
>
> DefaultSpeculator always add both MAP and REDUCE Speculative task even MAP_SPECULATIVE
or REDUCE_SPECULATIVE is disabled.
> If both MAP_SPECULATIVE and REDUCE_SPECULATIVE are disabled, 
> DefaultSpeculator won't start.
> The issue will happen if only one of MAP_SPECULATIVE and REDUCE_SPECULATIVE is enabled,
both MAP and REDUCE Speculative task  are generate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message