hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Bhattacharjee <rahul.rec....@gmail.com>
Subject Re: Question related to resource allocation in Yarn!
Date Fri, 06 Sep 2013 05:27:34 GMT
Hi Devaraj,

I am on Hadoop 2.0.4 . I am able to get containers now and my yarn app runs
properly. I am setting hostname as * , while requesting containers.

There is no problem as of now , only thing is I am allocated only 2
containers at one time , however I believe that the node manager can run
more containers.

2013-09-06 09:53:38,433 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerNode:
Assigned container container_1378441324025_0001_01_000001 of capacity
<memory:100, vCores:1> on host storyacid-lm:55407, which currently has 1
containers, <memory:100, vCores:1> used and <memory:8092, vCores:15>
available

I am requesting containers with 100 mb mem and 1 core. If  I know more
about how is the capacity if calculated per node , or how the allocation is
done , then it would be useful.

Thanks for the help!
Rahul







On Fri, Sep 6, 2013 at 10:31 AM, Devaraj k <devaraj.k@huawei.com> wrote:

>  Hi Rahul,****
>
> ** **
>
>     Could you tell me, what is the version you are using?****
>
>      ****
>
> **·         **If you want a container, you need to issue 3 resource
> requests (1-node local, 1-rack local and 1-Any(*) ). If you are using
> 2.1.0-beta<https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=ASC&sorter/field=priority&pid=12313722&customfield_12310320=12324029>or
later versions, you can set the Relax Locality flag to false for getting
> only on the specified host.****
>
> Can you also share the code how you are requesting for containers…so that
> we can help you better..****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* Rahul Bhattacharjee [mailto:rahul.rec.dgp@gmail.com]
> *Sent:* 06 September 2013 09:43
> *To:* user@hadoop.apache.org
> *Subject:* Re: Question related to resource allocation in Yarn!****
>
> ** **
>
> I could progress a bit on this.
>
> I was not setting responseId while asking for containers.****
>
> Still I have one question as why I am only been allocated two containers
> whereas node manager can run more containers.
>
> Response while registering the application master -
> AM registration response minimumCapability {, memory: 1024, virtual_cores:
> 1, }, maximumCapability {, memory: 8192, virtual_cores: 32, }, ****
>
> Thanks,****
>
> Rahul****
>
> ** **
>
> On Thu, Sep 5, 2013 at 8:33 PM, Rahul Bhattacharjee <
> rahul.rec.dgp@gmail.com> wrote:****
>
> Hi,****
>
> I am trying to make a small poc on top of yarn.****
>
> Within the launched application master , I am trying to request for 50
> containers and launch  a same task on those allocated containers.****
>
> My config : AM registration response minimumCapability {, memory: 1024,
> virtual_cores: 1, }, maximumCapability {, memory: 8192, virtual_cores: 32,
> }, ****
>
> 1) I am asking for 1G mem and 1 core container to the RM. Ideally the RM
> should return me 6 - 7 containers , but the response always returns with
> only 2 containers. ****
>
> Why is that ?****
>
> 2) So , when in the first ask 2 containers are returned , then I again
> required the RM for 50 - 2 = 48 containers. I keep getting 0 containers ,
> even if the previously started containers have finished.****
>
> Why is that ?****
>
> Any link explaining the allocate request of RM would be very helpful.****
>
> ** **
>
> Thanks,
> Rahul****
>
> ** **
>

Mime
View raw message