Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1273192339.1215672692419.JavaMail.jira@brutus>
Date: Wed, 9 Jul 2008 23:51:32 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3412) Refactor the scheduler out of the
 JobTracker
In-Reply-To: <53274895.1211066635619.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HADOOP-3412?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D126=
12388#action_12612388 ]=20

Owen O'Malley commented on HADOOP-3412:
---------------------------------------

It is unfortunate that this likely conflicts with HADOOP-3445, but this pat=
ch has received a lot more attention. (HADOOP-3445 isn't even patch availab=
le yet...) It sucks when large patches conflict, but it is hard to avoid on=
 large projects with a lot of contributors. We should push to finish review=
ing this patch to get it in, so that 3445 can use it as a base to make chan=
ges from.

> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
>                 Key: HADOOP-3412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3412
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Brice Arnould
>            Assignee: Brice Arnould
>            Priority: Minor
>             Fix For: 0.19.0
>
>         Attachments: JobScheduler.patch, JobScheduler_v2.patch, JobSchedu=
ler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch, JobScheduler_v=
5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.patch, JobScheduler_v6.=
3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch, JobScheduler_v7.1.=
patch, JobScheduler_v7.patch, JobScheduler_v8.patch, RackAwareJobScheduler.=
java, SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very nai=
ve. I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.=
 But, it is hard to implement and compare schedulers, because the schedulin=
g logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler.=
 It re-implements the current scheduling algorithm in a separate class call=
ed JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour th=
an the current JobTracker. More precisely, it doesn't re-implement things l=
ike code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current implementa=
tion, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you de=
cide about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step=
 of the scheduling. I hope that it will help future improvement by lowering=
 the level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (mak=
ing it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propos=
e a JobSchedulerInterface that many JobSchedulers could implement and to pr=
opose alternatives to the current =C2=AB FifoJobScheduler =C2=BB.  If some =
of you have ideas about that please tell ^^ I will also open issues for thi=
ngs marked as FIXME in the patch.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.