hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: Multiple resource requests for a given node (or all nodes)?
Date Tue, 13 Dec 2011 15:32:07 GMT
Arun,

I am saying that I don't know what the correct solution is to updating the scheduler interface.
 Perhaps the correct solution is no change, I have not taken the time to think about it much.
 What I am saying is that there are a number of new features that are likely going to be going
into the scheduler, and if we are going to change the interface, I want to be sure that we
think about these use cases before we change it.  That is all I am saying.  I am not advocating
for a particular interface at this point, as I said I have not taken the time to think about
it in depth.

--Bobby Evans

On 12/13/11 12:42 AM, "Arun C Murthy" <acm@hortonworks.com> wrote:

I'd argue that Robert is complaining that the interface *is not* MR-centric enough.

IAC, priorities is fairly generic. MR AM uses it to get constraints to stick.

Arun

On Dec 12, 2011, at 7:50 PM, Patrick Wendell wrote:

> Todd - that's a good question and I haven't looked closely into
> whether simply adding a multimap is enough or if there are more deeply
> seeded issues (at least to address this specific case). If it's the
> former I'll probably just submit a patch.
>
> Arun - that seems like a hack but I guess it is a sufficient
> workaround for current applications.
>
> I'm finishing up a bare-bones version of the Fair Scheduler right now
> (going to throw something up for review soon) but I haven't yet added
> preemption.  How this is going to work well with various types of
> applications is unclear. In the MR case we can probably just preempt
> based on priorities, since they are essentially just ordering
> constraints right now. As Robert points out, this interface is very
> MR-Centric right now - i'm not sure this generalizes well to other
> applications depending on how they use priorities.
>
> - Patrick
>
> On Mon, Dec 12, 2011 at 1:27 PM, Arun C Murthy <acm@hortonworks.com> wrote:
>> Use priorities to ask for different resource types.
>>
>> Arun
>>
>> On Dec 10, 2011, at 12:23 PM, Patrick Wendell wrote:
>>
>>> If you look at how resource requests are stored now, they use a map
>>> keyed on the node hostname.
>>>
>>> == AppSchedulingInfo.java ==
>>>
>>>  final Map<Priority, Map<String, ResourceRequest>> requests =
>>>    new HashMap<Priority, Map<String, ResourceRequest>>();
>>>
>>> ========
>>>
>>> What happens if an application wants to request multiple container
>>> types on a given node. E.g. say I need 10 2GB containers and 10 1GB
>>> containers, and I don't care which node they are on (i.e. RMNode.ANY).
>>> I really want to store 2 resource requests under RMNode.ANY in this
>>> case... don't I?
>>>
>>> Is the model just that an AM would ask for these in series?
>>>
>>> - Patrick
>>



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message