ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-3414) Hadoop: Optimize map-reduce job planning.
Date Tue, 19 Jul 2016 08:07:20 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383758#comment-15383758

Vladimir Ozerov commented on IGNITE-3414:

1) Because we want to assign local reducers until some limit is reached, and only then switch
to weight-based distribution. This way when there are only several reducers, they are likely
to be assigned to nodes where some mappers are run. If we rely only on weights, the same behavior
is only possible when local and remote reducer weights are different. And this will be bad
for cases when there are lots reducers - the imbalance between local and remote nodes will
grow linearly.
2) This is exactly how it works now. 
3) Too difficult for a very rare case when there are more than one node on a single machine.

> Hadoop: Optimize map-reduce job planning.
> -----------------------------------------
>                 Key: IGNITE-3414
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3414
>             Project: Ignite
>          Issue Type: Task
>          Components: hadoop
>    Affects Versions: 1.6
>            Reporter: Vladimir Ozerov
>            Assignee: Vladimir Ozerov
>            Priority: Critical
>             Fix For: 1.7
> Currently Hadoop module has inefficient map-reduce planning engine. In particular, it
assigns tasks only to affinity nodes. It could lead to situation when very huge tasks is processed
by a single cluster node, while other cluster nodes are idle. 
> We should implement configurable map-reduce planner which will be able to utilize the
whole cluster.

This message was sent by Atlassian JIRA

View raw message