Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 209.85.211.172 is neither permitted
 nor denied by domain of mnagendr@asu.edu)
MIME-Version: 1.0
In-Reply-To: <45f85f70908131052v6d6d547dtb291eacb8b42efa1@mail.gmail.com>
References: <77f4f8890908131044u38eed342o40fadf0050fe56b4@mail.gmail.com>
	 <45f85f70908131052v6d6d547dtb291eacb8b42efa1@mail.gmail.com>
Date: Thu, 13 Aug 2009 11:32:48 -0700
Message-ID: <77f4f8890908131132w77d93e25i494ccf55ff996224@mail.gmail.com>
Subject: Re: Intermediary Data on Fair Scheduler
From: Mithila Nagendra <mnagendr@asu.edu>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e642dd649b51b504710a29c7

--0016e642dd649b51b504710a29c7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Todd

So does this mean that when two jobs are assigned to a pool, where one job
has 1 map task and 1 reduce task, whereas the other has 5 map and 5 reduce
tasks, how will the switch between these jobs take place?

Lets say the scheduler starts with the bigger job, runs 1 map task, when it
switches to the shorter job what does it do with the intermediate data? for
instance in Hadoop on demand if we run a search query where would the search
keywords be stored? I assume if the bigger job is in middle of a map task
the smaller job will wait for the task to end before the the map task for
the shorter job is launched.

Thanks!
Mithila

On Thu, Aug 13, 2009 at 10:52 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hi Mithila,
>
> I assume you're referring to fair scheduler preemption. In the preemption
> scenario, tasks are completely killed, not paused. It's not like a
> preemptive scheduler in your OS where things are "context switched". This
> is
> why the preemption is not enabled by default and has tuning parameters that
> only trigger preemption in certain situations.
>
> Hope that helps,
> -Todd
>
> On Thu, Aug 13, 2009 at 10:44 AM, Mithila Nagendra <mnagendr@asu.edu>
> wrote:
>
> > Hello All
> >
> > When the fair scheduler switches between two jobs, what does it do with
> the
> > intermediary data? Does it dump the data/job states onto the disk (DFS)?
> Or
> > does it do a context switch (i.e. everything is in memory)? I was looking
> > at
> > the scheduler for an application I'm working on, any pointers will be
> > appreciated!
> >
> > Thanks!
> > Mithila Nagendra
> > Arizona State University
> >
>

--0016e642dd649b51b504710a29c7--