Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <785153156.1216042956419.JavaMail.jira@brutus>
Date: Mon, 14 Jul 2008 06:42:36 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Updated: (HADOOP-3412) Refactor the scheduler out of the
 JobTracker
In-Reply-To: <53274895.1211066635619.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HADOOP-3412?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated HADOOP-3412:
------------------------------

    Status: Patch Available  (was: Open)

> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
>                 Key: HADOOP-3412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3412
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Brice Arnould
>            Assignee: Brice Arnould
>            Priority: Minor
>             Fix For: 0.19.0
>
>         Attachments: JobScheduler-v10.patch, JobScheduler-v9.1.patch, Job=
Scheduler-v9.2.patch, JobScheduler-v9.patch, JobScheduler.patch, JobSchedul=
er_v2.patch, JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4=
.patch, JobScheduler_v5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.p=
atch, JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.pat=
ch, JobScheduler_v7.1.patch, JobScheduler_v7.patch, JobScheduler_v8.patch, =
RackAwareJobScheduler.java, SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very nai=
ve. I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.=
 But, it is hard to implement and compare schedulers, because the schedulin=
g logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler.=
 It re-implements the current scheduling algorithm in a separate class call=
ed JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour th=
an the current JobTracker. More precisely, it doesn't re-implement things l=
ike code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current implementa=
tion, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you de=
cide about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step=
 of the scheduling. I hope that it will help future improvement by lowering=
 the level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (mak=
ing it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propos=
e a JobSchedulerInterface that many JobSchedulers could implement and to pr=
opose alternatives to the current =C2=AB FifoJobScheduler =C2=BB.  If some =
of you have ideas about that please tell ^^ I will also open issues for thi=
ngs marked as FIXME in the patch.

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.