hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Q <bill.q....@gmail.com>
Subject Re: Issues with ResourceManager scheduling functions
Date Mon, 09 Dec 2013 08:59:45 GMT
Thanks a lot.

So, in order for the two types of containers mentioned above to work, what
priorities should I assign to them? Can you give an example?

Many thanks.


Bill


On Mon, Dec 9, 2013 at 3:30 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

> YARN currently is unable to handle requests with different resource
> requirements at the same priority (YARN-314).  Using different priorities
> would likely solve the issue.  I believe it would also be safest to avoid
> using the priority 0 because this is used for the application master.
>
> -Sandy
>
>
> On Mon, Dec 9, 2013 at 12:23 AM, Bill Q <bill.q.hdp@gmail.com> wrote:
>
>> No, all default 0. Should I use different priorities?
>>
>> Many thanks.
>>
>>
>> Bill
>>
>>
>> On Mon, Dec 9, 2013 at 2:53 AM, Sandy Ryza <sandy.ryza@cloudera.com>wrote:
>>
>>> Are you requesting the different container types at different priorities?
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Dec 8, 2013 at 8:52 PM, Bill Q <bill.q.hdp@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I recently implemented an application using YARN. Everything works
>>>> fine, except for the illogical scheduling behavior of the ResourceManger,
>>>> which might be caused by the heterogeneity of the cluster.
>>>>
>>>> I have two types of computing node in the cluster. 10 nodes with 128G
>>>> memory and 10 nodes with 64G memory. In the application, I create two types
>>>> of container. Type A container would ask for 50G of memory and Type B
>>>> container would ask for 100G of memory.
>>>>
>>>> The problem is that if I created 50 Type B containers (100G), and 50
>>>> Type A containers (50G), Only Type B container (100G) are running. Instead
>>>> of running the Type A container (50G), all the nodes with 64G memory are
>>>> idling. And only the nodes with 128G memory are running Type B container
>>>> (100G). This doesn't make too much sense to me.
>>>>
>>>> If I only created Type A containers (50G), all the nodes would be able
>>>> run the containers. The 128G node would run two containers simultaneously.
>>>> So, the problem seems to be connected to the different types of containers.
>>>>
>>>> Does anybody have any suggestions to fix this issue?
>>>>
>>>> Many thanks.
>>>>
>>>>
>>>> Bill
>>>>
>>>
>>>
>>
>

Mime
View raw message