hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gaurav <gau...@datatorrent.com>
Subject Re: Allocating Containers on a particular Node in Yarn
Date Thu, 14 Nov 2013 01:30:12 GMT
Hi,

I looked in the trunk and line numbers are 361 and 366.

Thanks
-Gaurav

On 11/13/2013 5:04 PM, gaurav wrote:
> I have hadoop-2.2.0
> Thanks
> -Gaurav
> On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>> What version are you using?  Setting the relax locality to true for 
>> nodes is always correct.  For racks, this is not necessarily the 
>> case.  When I look at trunk, it is set to true always on line 361 
>> (which is correct), but on on line 374.
>>
>> -Sandy
>>
>>
>> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <gaurav@datatorrent.com 
>> <mailto:gaurav@datatorrent.com>> wrote:
>>
>>     Hi Sandy,
>>
>>     No it is not working for me.  As mentioned earlier, AMRMClient is
>>     not respecting the locality for node and rack in *Line 361 and
>>     374 *and is set to true always.
>>     I am requesting for one container with specified node and rack
>>     and relaxed locality false.
>>
>>     Thanks
>>     -Gaurav
>>
>>     On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>>     [moving to user list]
>>>
>>>     Right.  relaxLocality needs to be set on the next level up.  It
>>>     determines whether locality can be relaxed to that level.
>>>      Confusing, I know.  If you are using AMRMClient, you should be
>>>     able to accomplish what you're looking for by creating an
>>>     AMRMClient.ContainerRequest that specifies a node and sets relax
>>>     locality to false.  Is that not working for you?
>>>
>>>     -Sandy
>>>
>>>
>>>     On Wed, Nov 13, 2013 at 4:19 PM, gaurav <gaurav@datatorrent.com
>>>     <mailto:gaurav@datatorrent.com>> wrote:
>>>
>>>         Hi Sandy,
>>>
>>>         If I set relaxlocality to true with host name, I don't get
>>>         the container on specified host even though node has the
>>>         resources.
>>>         I am using AMRMClient, only thing is I made following
>>>         changes to get the containers on the specified node.
>>>         /
>>>         //Line: 361 //
>>>         //Original//
>>>         // addResourceRequest(req.getPriority(), node,
>>>         req.getCapability(), req, true);//
>>>         //Modifiled//
>>>         // addResourceRequest(req.getPriority(), node,
>>>         req.getCapability(), req, req.getRelaxLocality());//
>>>         //
>>>         //Line 374//
>>>         //Original//
>>>         // addResourceRequest(req.getPriority(), rack,
>>>         req.getCapability(), req, true);//
>>>         //Modifiled//
>>>         // addResourceRequest(req.getPriority(), rack,
>>>         req.getCapability(), req, req.getRelaxLocality());//
>>>         ///
>>>
>>>
>>>         Thanks
>>>         -Gaurav
>>>
>>>
>>>         On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>>>
>>>             Yeah, specifying a host name with relaxLocality is
>>>             meaningful.  Schedulers
>>>             use delay scheduling (
>>>             http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf
>>>             <http://www.cs.berkeley.edu/%7Ematei/talks/2010/eurosys_delaysched.pdf>)
>>>             to
>>>             achieve locality when relaxLocality is on.  But it is
>>>             turned off by
>>>             default.  The individual scheduler docs have specifics
>>>             on how to configure
>>>             it.
>>>
>>>             Guarav,
>>>             Using ResourceRequests directly is not straightforward
>>>             and error prone.  Is
>>>             there a reason that AMRMClient is unsuitable for your needs?
>>>
>>>             -Sandy
>>>
>>>
>>>
>>>             On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise
>>>             <thomas.weise@gmail.com
>>>             <mailto:thomas.weise@gmail.com>>wrote:
>>>
>>>                 Is it possible to specify a particular node and have
>>>                 RM fallback to an
>>>                 different node only after making an attempt to
>>>                 allocate for the requested
>>>                 node? In other words, is the combination of specific
>>>                 host name and
>>>                 relaxLocality=TRUE meaningful at all?
>>>
>>>                 Thanks.
>>>
>>>
>>>                 On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur
>>>                 <tucu@cloudera.com <mailto:tucu@cloudera.com>
>>>
>>>                     wrote:
>>>                     Gaurav,
>>>
>>>                     Setting relaxLocality to FALSE should do it.
>>>
>>>                     thanks.
>>>
>>>
>>>                     On Wed, Nov 13, 2013 at 2:58 PM, gaurav
>>>                     <gaurav@datatorrent.com
>>>                     <mailto:gaurav@datatorrent.com>> wrote:
>>>
>>>                           Hi,
>>>                           I am trying to allocate containers on a
>>>                         particular node in Yarn but
>>>
>>>                 Yarn
>>>
>>>                         is returning me containers on different node
>>>                         although the requested
>>>
>>>                 node
>>>
>>>                         has resources available.
>>>
>>>                         I checked into the allocate(AllocateRequest
>>>                         request) function of
>>>                         ApplicationMasterService and my request is
>>>                         as follows
>>>
>>>                         *request: ask { priority { priority: 1 }
>>>                         resource_name: "h2"
>>>
>>>                 capability {
>>>
>>>                         memory: 1000 } num_containers: 2 } ask {
>>>                         priority { priority: 1 }
>>>                         resource_name: "/default-rack" capability {
>>>                         memory: 1000 }
>>>
>>>                     num_containers:
>>>
>>>                         2 } ask { priority { priority: 1 }
>>>                         resource_name: "*" capability {
>>>
>>>                     memory:
>>>
>>>                         1000 } num_containers: 2 } response_id: 1
>>>                         progress: 0.0*
>>>
>>>                         but the containers that I am getting back is
>>>                         as follows
>>>                         [Container: [ContainerId:
>>>                         container_1384381084244_0001_01_000002,
>>>
>>>                 NodeId:
>>>
>>>                         h1:1234, NodeHttpAddress: h1:2, Resource:
>>>                         <memory:1024, vCores:1>,
>>>                         Priority: 1, Token: Token { kind:
>>>                         ContainerToken, service: h1:1234 },
>>>
>>>                 ],
>>>
>>>                         Container: [ContainerId:
>>>                         container_1384381084244_0001_01_000003,
>>>
>>>                 NodeId:
>>>
>>>                         h1:1234, NodeHttpAddress: h1:2, Resource:
>>>                         <memory:1024, vCores:1>,
>>>                         Priority: 1, Token: Token { kind:
>>>                         ContainerToken, service: h1:1234 },
>>>
>>>                 ]]
>>>
>>>                         I am attaching the test case that I have
>>>                         written along with the mail.
>>>
>>>                 It
>>>
>>>                         uses classes under
>>>                         org.apache.hadoop.yarn.server.resourcemanager
>>>
>>>                 package.
>>>
>>>                         Any pointers would be of great help
>>>
>>>                         Thanks
>>>                         Gaurav
>>>
>>>
>>>
>>>
>>>
>>>
>>>                     --
>>>                     Alejandro
>>>
>>>
>>>
>>
>>
>


Mime
View raw message