hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-944) Extend FairShare scheduler to fair-share memory usage in the cluster
Date Mon, 07 Sep 2009 11:47:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752105#action_12752105
] 

Vinod K V commented on MAPREDUCE-944:
-------------------------------------

I see in the patch attached that only one concrete implementation CapBasedLoadManager is done
for the LoadManager which in turn doesn't take into account any resource usage. I guess you
are planning a proper implementation for this feature regarding fair-share of memory usage
in another JIRA.

Some points still not dealt with in this JIRA. I bring about these points so as to know if
you are thinking or have already thought anything about this.
 - Job configuration about how users specify the resource usage. Some memory related configuration
properties are added to the framework while working for memory monitoring on TTs as well as
memory usage based scheduling in CapacityTaskScheduler. You may want to reuse some/all of
it.
 - Capturing the scheduling decisions involved when we are not able to find a task from a
Schedulable because of lack of resources on a given TaskTasker.

Regarding the latter, the current patch just returns null, which is similar to the decision
CapacityTaskScheduler used to take in previous versions - i.e. block the TT till it can be
given a task from the job at the head of the queue/pool. Sometime back, we investigated how
this approach works with FairScheduler and realized some important implications. For e.g,
because the order of jobs might change significantly in consecutive iterations of FairScheduler,
just returning null may not work at all. Eventually we may end up waiting for a long time
if significant number of jobs ask for high amount of resources.

Thoughts?

> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
>                 Key: MAPREDUCE-944
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>         Attachments: LoadManager.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate allocating new
tasks on a particular TaskTracker. In similar lines, it would be nice if the FairShare Scheduler
can have a pluggable policy to regulate new tasks from a particular job. This will allow one
to skip scheduling tasks of a job that  is eating a large percentage of memory in the cluster,
i.e. fair-share of memory resources among jobs. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message