Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of yhemanth@gmail.com designates
 209.85.160.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=PxsdcR8nVTwlvK2FSrH9Ni0eeCi7El2SEMq92buBU11aMKJeKHIV5DqSaTpQH3TPyr
         E/jds7ll4nnRQsdnvBh+Gxw1J8MlhlaXMIf6VzMLnONkDX91k3Ps31sEv+oXT51TNnu6
         hiOri5JI3ySiklIJWbeGL8OavHqw5mmY/TUXA=
MIME-Version: 1.0
In-Reply-To: <AANLkTikbjKABc-b0XujDa1SnW2SPFDQUWWP011O6KqTn@mail.gmail.com>
References: <AANLkTik9+-vyLkXQMkDm7hdPq+1L7pwaYenjQLSY3EJJ@mail.gmail.com>
	<AANLkTimtfFD9MfA361USL_+bYY+1Yu-xBViKFz8is6fh@mail.gmail.com>
	<AANLkTimU9RJ-F_kQz4t=0FceQioNiC_DACSOBs56zXw=@mail.gmail.com>
	<AANLkTineTdvzFgqtgrXzNQTa8ano97HVSY+PyLX6qupg@mail.gmail.com>
	<AANLkTikbjKABc-b0XujDa1SnW2SPFDQUWWP011O6KqTn@mail.gmail.com>
Date: Sat, 23 Oct 2010 08:42:44 +0530
Message-ID: <AANLkTik78k2qezmfFm_hewbE1=krt7WasBDkVYAJYNj6@mail.gmail.com>
Subject: Re: Hadoop's deafult FIFO scheduler
From: Hemanth Yamijala <yhemanth@gmail.com>
To: common-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,

Sorry for a delayed response.

Once jobs are submitted, they are setup by running the setup task.
These are run in order of submission. However, the setup task itself
runs on any free map or reduce slot on any node. I can imagine
scenarios where the setup task of a job that was submitted later
completes first. And when that happens, it can start running out of
order.

Thanks
Hemanth

On Fri, Oct 15, 2010 at 3:41 PM, He Chen <airbots@gmail.com> wrote:
> Hi Hemanth
>
> all jobs were submitted within a minute and a few
> seconds between jobs. The hadoop version is 0.20.2
>
> Thanks
>
>
> On Thu, Oct 14, 2010 at 11:45 PM, Hemanth Yamijala <yhemanth@gmail.com>wr=
ote:
>
>> Hi,
>>
>> On Thu, Oct 14, 2010 at 10:59 PM, He Chen <airbots@gmail.com> wrote:
>> > they arrived in 1 minute. I understand there will be a setup phase whi=
ch
>> > will use any free slot no matter map or reduce.
>> >
>>
>> You mean all jobs were submitted within a minute ? That means a few
>> seconds between jobs ? Or do you mean each job was submitted a minute
>> after the earlier job. Also, which version of Hadoop is this ?
>>
>> > My queue time is the period between the start of Map stage and the tim=
e
>> job
>> > is submitted. Because the setup phase has the higher priority than map
>> and
>> > reduce tasks. Any job submitted in the queue will setup no matter how
>> many
>> > previous map and reduce tasks need to be assigned.
>> >
>> > Now, I am sure the job3 setup stage finished earlier than job4's.
>> However,
>> > job3's map stage start later than job4's. BTW, they request same amoun=
t
>> of
>> > blocks.
>> >
>> >
>> > On Thu, Oct 14, 2010 at 12:10 PM, abhishek sharma <absharma@usc.edu>
>> wrote:
>> >
>> >> What is the inter-arrival time between these jobs?
>> >>
>> >> There is a "set up" phase for jobs before they are launched. It is
>> >> possible that the order of jobs can change due to slightly different
>> >> set up times. Apart from the number of blocks, it may matter "where"
>> >> these blocks lie.
>> >>
>> >> Abhishek
>> >>
>> >> On Thu, Oct 14, 2010 at 10:06 AM, He Chen <airbots@gmail.com> wrote:
>> >> > Hi all
>> >> >
>> >> > I am testing the performance of my Hadoop clsuters with Hadoop Defa=
ult
>> >> FIFO
>> >> > schedular. But I find a interesting phenomina.
>> >> >
>> >> > When I submit a series of jobs, some job will be executed earlier e=
ven
>> >> they
>> >> > are submitted late. All jobs are request same amount of blocks. For
>> >> example:
>> >> > job 1 =A0submit at time 0
>> >> > job 2 submit at time 1
>> >> > job 3 submit at time 2
>> >> > job 4 submit at time 3
>> >> >
>> >> >
>> >> > job 4 's queue time is smaller than job3's queue time. This disobey
>> the
>> >> FIFO
>> >> > principle. Any one can give a hint?
>> >> >
>> >> > Thanks
>> >> >
>> >> > Chen
>> >> >
>> >>
>> >
>>
>