hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: preferential distribution of tasks to tasktracker by jobtracker based on specific criteria (cpu frequency)
Date Thu, 07 Mar 2013 01:00:01 GMT
MRv1: Both the CapacityScheduler and FairScheduler support some form
of resource aware scheduling (Capacity supports memory provisions
while Fair provides an interface you can plug into to influence its
decisions). You can also implement a custom scheduler - as
TaskTrackers do send their resource details (which includes CPU). The
class you need to derive for this is
and the assignTasks(…)'s TaskTracker object (a repr. of the TT for the
JT) lets you get the TaskTrackerStatus object which has the relevant
information you seek:

Note though that in MRv2, which runs on YARN and is present in the
current 2.x releases, the JobTracker and the TaskTrackers are both
gone and the scheduler is now a part of the Resource Manager as a
generic component under YARN. Just something to keep in mind if you're
continuing to base your work (research or otherwise) on an older,
maintenance release. Your MRv1 implementation may become irrelevant or
cease to work after the upgrade in future.

YARN: The JIRA https://issues.apache.org/jira/browse/YARN-2 added
native CPU core based scheduling, which should also help your need
right out of the box perhaps.

On Thu, Mar 7, 2013 at 2:17 AM, Sayan Kole <sayankole@gmail.com> wrote:
> Hi,
>    I want the jobtracker to prioritize the assignment of tasks to certain
> tasktrackers.
> eg: If a tasktracker meets certain criteria better than other ones, I want
> to assign task to that tasktracker first (ideally I want the jobtracker to
> sort tasktrackers based on certain criteria (eg cpu frequency) and then
> assign tasks to tasktracker based on that sorted assignment.
> Could you point me to the files(eg Jobtracker.java....) and routines to
> modify or a guideline to a proper way of implementing it.
> Thanks,
> Sayan Kole.

Harsh J

View raw message