hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2812) Bias the decision of task scheduling (both for not-running and running) on node metrics (load, processing rate etc).
Date Wed, 13 Feb 2008 06:42:09 GMT
Bias the decision of task scheduling (both for not-running and running) on node metrics (load,
processing rate etc).
--------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-2812
                 URL: https://issues.apache.org/jira/browse/HADOOP-2812
             Project: Hadoop Core
          Issue Type: Improvement
          Components: mapred
            Reporter: Amar Kamat


HADOOP-2014 deals with a similar issue but at the rack level. This issue deals with the nodes
on the same rack. Consider the following case
||node/host||Runnable && ~Running TIPs||host status||
|H1|T1|good|
|H2|T2|good|
|H3|T3|good|
|H4|T4,T5,T6|bad|
|H5|-|good|
|rack1|T1,T2,T3,T4,T5,T6|good in all terms (runnable/numSlots) etc|
Now if H5 asks for a new task to run,  it will be given T1. Ideally it should be given T4
because the processing rate of H1 is better than H4. Giving T1 will kill the locality of a
better node. It makes more sense to kill the locality of a bad node instead. If H4 is overloaded
then its better to select T4 since that might increase the chances of hitting the locality.
With HADOOP-2014 the chances of TIPs from rack1 getting selected decreases since there might
be other racks which are overloaded and the hosts having T4,T5,T6 (locally) are in better
shape than H5. The point is select splits from nodes (rack local) that either have many local
splits or might not be able to process them faster. 
----
Even for speculation, its better to speculate a TIP that _might_ take longer than expected
amongst those which can be speculated. Consider the following
||node/host||Running TIPs||num TIPs processed|host status||
|H1|T1|2|good|
|H2|T2|1|bad|
Here it makes sense to speculate T2 since there is a reason to believe that T2 on H2 might
become a long tail.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message