hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gaurav <gau...@datatorrent.com>
Subject Re: Allocating Containers on a particular Node in Yarn
Date Thu, 14 Nov 2013 01:04:48 GMT
I have hadoop-2.2.0

Thanks
-Gaurav

On 11/13/2013 4:59 PM, Sandy Ryza wrote:
> What version are you using?  Setting the relax locality to true for 
> nodes is always correct.  For racks, this is not necessarily the case. 
>  When I look at trunk, it is set to true always on line 361 (which is 
> correct), but on on line 374.
>
> -Sandy
>
>
> On Wed, Nov 13, 2013 at 4:47 PM, gaurav <gaurav@datatorrent.com 
> <mailto:gaurav@datatorrent.com>> wrote:
>
>     Hi Sandy,
>
>     No it is not working for me.  As mentioned earlier, AMRMClient is
>     not respecting the locality for node and rack in *Line 361 and 374
>     *and is set to true always.
>     I am requesting for one container with specified node and rack and
>     relaxed locality false.
>
>     Thanks
>     -Gaurav
>
>     On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>     [moving to user list]
>>
>>     Right.  relaxLocality needs to be set on the next level up.  It
>>     determines whether locality can be relaxed to that level.
>>      Confusing, I know.  If you are using AMRMClient, you should be
>>     able to accomplish what you're looking for by creating an
>>     AMRMClient.ContainerRequest that specifies a node and sets relax
>>     locality to false.  Is that not working for you?
>>
>>     -Sandy
>>
>>
>>     On Wed, Nov 13, 2013 at 4:19 PM, gaurav <gaurav@datatorrent.com
>>     <mailto:gaurav@datatorrent.com>> wrote:
>>
>>         Hi Sandy,
>>
>>         If I set relaxlocality to true with host name, I don't get
>>         the container on specified host even though node has the
>>         resources.
>>         I am using AMRMClient, only thing is I made following changes
>>         to get the containers on the specified node.
>>         /
>>         //Line: 361 //
>>         //Original//
>>         // addResourceRequest(req.getPriority(), node,
>>         req.getCapability(), req, true);//
>>         //Modifiled//
>>         // addResourceRequest(req.getPriority(), node,
>>         req.getCapability(), req, req.getRelaxLocality());//
>>         //
>>         //Line 374//
>>         //Original//
>>         // addResourceRequest(req.getPriority(), rack,
>>         req.getCapability(), req, true);//
>>         //Modifiled//
>>         // addResourceRequest(req.getPriority(), rack,
>>         req.getCapability(), req, req.getRelaxLocality());//
>>         ///
>>
>>
>>         Thanks
>>         -Gaurav
>>
>>
>>         On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>>
>>             Yeah, specifying a host name with relaxLocality is
>>             meaningful.  Schedulers
>>             use delay scheduling (
>>             http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf
>>             <http://www.cs.berkeley.edu/%7Ematei/talks/2010/eurosys_delaysched.pdf>)
>>             to
>>             achieve locality when relaxLocality is on.  But it is
>>             turned off by
>>             default.  The individual scheduler docs have specifics on
>>             how to configure
>>             it.
>>
>>             Guarav,
>>             Using ResourceRequests directly is not straightforward
>>             and error prone.  Is
>>             there a reason that AMRMClient is unsuitable for your needs?
>>
>>             -Sandy
>>
>>
>>
>>             On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise
>>             <thomas.weise@gmail.com
>>             <mailto:thomas.weise@gmail.com>>wrote:
>>
>>                 Is it possible to specify a particular node and have
>>                 RM fallback to an
>>                 different node only after making an attempt to
>>                 allocate for the requested
>>                 node? In other words, is the combination of specific
>>                 host name and
>>                 relaxLocality=TRUE meaningful at all?
>>
>>                 Thanks.
>>
>>
>>                 On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur
>>                 <tucu@cloudera.com <mailto:tucu@cloudera.com>
>>
>>                     wrote:
>>                     Gaurav,
>>
>>                     Setting relaxLocality to FALSE should do it.
>>
>>                     thanks.
>>
>>
>>                     On Wed, Nov 13, 2013 at 2:58 PM, gaurav
>>                     <gaurav@datatorrent.com
>>                     <mailto:gaurav@datatorrent.com>> wrote:
>>
>>                           Hi,
>>                           I am trying to allocate containers on a
>>                         particular node in Yarn but
>>
>>                 Yarn
>>
>>                         is returning me containers on different node
>>                         although the requested
>>
>>                 node
>>
>>                         has resources available.
>>
>>                         I checked into the allocate(AllocateRequest
>>                         request) function of
>>                         ApplicationMasterService and my request is as
>>                         follows
>>
>>                         *request: ask { priority { priority: 1 }
>>                         resource_name: "h2"
>>
>>                 capability {
>>
>>                         memory: 1000 } num_containers: 2 } ask {
>>                         priority { priority: 1 }
>>                         resource_name: "/default-rack" capability {
>>                         memory: 1000 }
>>
>>                     num_containers:
>>
>>                         2 } ask { priority { priority: 1 }
>>                         resource_name: "*" capability {
>>
>>                     memory:
>>
>>                         1000 } num_containers: 2 } response_id: 1
>>                         progress: 0.0*
>>
>>                         but the containers that I am getting back is
>>                         as follows
>>                         [Container: [ContainerId:
>>                         container_1384381084244_0001_01_000002,
>>
>>                 NodeId:
>>
>>                         h1:1234, NodeHttpAddress: h1:2, Resource:
>>                         <memory:1024, vCores:1>,
>>                         Priority: 1, Token: Token { kind:
>>                         ContainerToken, service: h1:1234 },
>>
>>                 ],
>>
>>                         Container: [ContainerId:
>>                         container_1384381084244_0001_01_000003,
>>
>>                 NodeId:
>>
>>                         h1:1234, NodeHttpAddress: h1:2, Resource:
>>                         <memory:1024, vCores:1>,
>>                         Priority: 1, Token: Token { kind:
>>                         ContainerToken, service: h1:1234 },
>>
>>                 ]]
>>
>>                         I am attaching the test case that I have
>>                         written along with the mail.
>>
>>                 It
>>
>>                         uses classes under
>>                         org.apache.hadoop.yarn.server.resourcemanager
>>
>>                 package.
>>
>>                         Any pointers would be of great help
>>
>>                         Thanks
>>                         Gaurav
>>
>>
>>
>>
>>
>>
>>                     --
>>                     Alejandro
>>
>>
>>
>
>


Mime
View raw message