hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: Allocating Containers on a particular Node in Yarn
Date Thu, 14 Nov 2013 03:36:26 GMT
In that case, the AMRMClient code looks correct to me.  Can you share the
code you've written against it that's not receiving the correct containers?


On Wed, Nov 13, 2013 at 5:30 PM, gaurav <gaurav@datatorrent.com> wrote:

>  Hi,
>
> I looked in the trunk and line numbers are 361 and 366.
>
> Thanks
> -Gaurav
>
> On 11/13/2013 5:04 PM, gaurav wrote:
>
> I have hadoop-2.2.0
>
> Thanks
> -Gaurav
>
> On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>
> What version are you using?  Setting the relax locality to true for nodes
> is always correct.  For racks, this is not necessarily the case.  When I
> look at trunk, it is set to true always on line 361 (which is correct), but
> on on line 374.
>
>  -Sandy
>
>
> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <gaurav@datatorrent.com> wrote:
>
>>  Hi Sandy,
>>
>> No it is not working for me.  As mentioned earlier, AMRMClient is not
>> respecting the locality for node and rack in *Line 361 and 374 *and is
>> set to true always.
>> I am requesting for one container with specified node and rack and
>> relaxed locality false.
>>
>> Thanks
>> -Gaurav
>>
>>  On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>
>>  [moving to user list]
>>
>>  Right.  relaxLocality needs to be set on the next level up.  It
>> determines whether locality can be relaxed to that level.  Confusing, I
>> know.  If you are using AMRMClient, you should be able to accomplish what
>> you're looking for by creating an AMRMClient.ContainerRequest that
>> specifies a node and sets relax locality to false.  Is that not working for
>> you?
>>
>>  -Sandy
>>
>>
>> On Wed, Nov 13, 2013 at 4:19 PM, gaurav <gaurav@datatorrent.com> wrote:
>>
>>> Hi Sandy,
>>>
>>> If I set relaxlocality to true with host name, I don't get the container
>>> on specified host even though node has the resources.
>>> I am using AMRMClient, only thing is I made following changes to get the
>>> containers on the specified node.
>>> /
>>> //Line: 361 //
>>> //Original//
>>> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
>>> true);//
>>> //Modifiled//
>>> // addResourceRequest(req.getPriority(), node, req.getCapability(), req,
>>> req.getRelaxLocality());//
>>> //
>>> //Line 374//
>>> //Original//
>>> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
>>> true);//
>>> //Modifiled//
>>> // addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
>>> req.getRelaxLocality());//
>>> ///
>>>
>>>
>>> Thanks
>>> -Gaurav
>>>
>>>
>>> On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>>>
>>>> Yeah, specifying a host name with relaxLocality is meaningful.
>>>>  Schedulers
>>>> use delay scheduling (
>>>> http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf) to
>>>> achieve locality when relaxLocality is on.  But it is turned off by
>>>> default.  The individual scheduler docs have specifics on how to
>>>> configure
>>>> it.
>>>>
>>>> Guarav,
>>>> Using ResourceRequests directly is not straightforward and error prone.
>>>>  Is
>>>> there a reason that AMRMClient is unsuitable for your needs?
>>>>
>>>> -Sandy
>>>>
>>>>
>>>>
>>>> On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise <thomas.weise@gmail.com
>>>> >wrote:
>>>>
>>>>  Is it possible to specify a particular node and have RM fallback to an
>>>>> different node only after making an attempt to allocate for the
>>>>> requested
>>>>> node? In other words, is the combination of specific host name and
>>>>> relaxLocality=TRUE meaningful at all?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>>
>>>>>> wrote:
>>>>>> Gaurav,
>>>>>>
>>>>>> Setting relaxLocality to FALSE should do it.
>>>>>>
>>>>>> thanks.
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 13, 2013 at 2:58 PM, gaurav <gaurav@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>>    Hi,
>>>>>>>   I am trying to allocate containers on a particular node in
Yarn but
>>>>>>>
>>>>>>  Yarn
>>>>>
>>>>>>  is returning me containers on different node although the requested
>>>>>>>
>>>>>>  node
>>>>>
>>>>>>  has resources available.
>>>>>>>
>>>>>>> I checked into the allocate(AllocateRequest request) function
of
>>>>>>> ApplicationMasterService and my request is as follows
>>>>>>>
>>>>>>> *request: ask { priority { priority: 1 } resource_name: "h2"
>>>>>>>
>>>>>>  capability {
>>>>>
>>>>>>  memory: 1000 } num_containers: 2 } ask { priority { priority: 1
}
>>>>>>> resource_name: "/default-rack" capability { memory: 1000 }
>>>>>>>
>>>>>> num_containers:
>>>>>>
>>>>>>> 2 } ask { priority { priority: 1 } resource_name: "*" capability
{
>>>>>>>
>>>>>> memory:
>>>>>>
>>>>>>> 1000 } num_containers: 2 } response_id: 1 progress: 0.0*
>>>>>>>
>>>>>>> but the containers that I am getting back is as follows
>>>>>>> [Container: [ContainerId: container_1384381084244_0001_01_000002,
>>>>>>>
>>>>>>  NodeId:
>>>>>
>>>>>>  h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>>>>>>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234
},
>>>>>>>
>>>>>>  ],
>>>>>
>>>>>>  Container: [ContainerId: container_1384381084244_0001_01_000003,
>>>>>>>
>>>>>>  NodeId:
>>>>>
>>>>>>  h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
>>>>>>> Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234
},
>>>>>>>
>>>>>>  ]]
>>>>>
>>>>>>  I am attaching the test case that I have written along with the
mail.
>>>>>>>
>>>>>>  It
>>>>>
>>>>>>  uses classes under org.apache.hadoop.yarn.server.resourcemanager
>>>>>>>
>>>>>>  package.
>>>>>
>>>>>>  Any pointers would be of great help
>>>>>>>
>>>>>>> Thanks
>>>>>>> Gaurav
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>>
>>>
>>
>>
>
>
>

Mime
View raw message