hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Murthy <...@hortonworks.com>
Subject Re: Scheduler functionality in MRv2
Date Sat, 26 Nov 2011 22:25:09 GMT
Praveen,

There are many ways to prevent what you described...

I'm in the process of adding more docs, for now pls take a look at the
following older blog post for more details:

http://developer.yahoo.com/blogs/hadoop/posts/2011/03/mapreduce-nextgen-scheduler/

Arun

Sent from my iPhone

On Nov 25, 2011, at 11:55 PM, Praveen Sripati <praveensripati@gmail.com> wrote:

> Hi,
>
> Let's consider the following scenario
>
> -> The MR Job has an InputSplit on host h1 and h2
> -> AM makes a request to the Scheduler for a container on h1 and h2
> -> The scheduler responds with containers c1 and c2 on h1 and h2
> -> But the AM uses c1 and releases c2 after 15 minutes
>
> In this scenario c2 cannot be allocated to any of the AM by the scheduler
> for that span of 15 minutes and the c2 remains idle. And the released
> container is also not accounted into the Queues quota. How can such
> scenario be avoided?
>
> Also, it would be nice to specify a more complex requests to the scheduler,
> like a request for (h1 or h2), (r1 or r2). Add some dependencies between
> the requests.
>
> Thanks,
> Praveen

Mime
View raw message