hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gaurav <gau...@datatorrent.com>
Subject Re: Allocating Containers on a particular Node in Yarn
Date Thu, 14 Nov 2013 00:47:23 GMT
Hi Sandy,

No it is not working for me.  As mentioned earlier, AMRMClient is not 
respecting the locality for node and rack in *Line 361 and 374 *and is 
set to true always.
I am requesting for one container with specified node and rack and 
relaxed locality false.

Thanks
-Gaurav

On 11/13/2013 4:24 PM, Sandy Ryza wrote:
> [moving to user list]
>
> Right.  relaxLocality needs to be set on the next level up.  It 
> determines whether locality can be relaxed to that level.  Confusing, 
> I know.  If you are using AMRMClient, you should be able to accomplish 
> what you're looking for by creating an AMRMClient.ContainerRequest 
> that specifies a node and sets relax locality to false.  Is that not 
> working for you?
>
> -Sandy
>
>
> On Wed, Nov 13, 2013 at 4:19 PM, gaurav <gaurav@datatorrent.com 
> <mailto:gaurav@datatorrent.com>> wrote:
>
>     Hi Sandy,
>
>     If I set relaxlocality to true with host name, I don't get the
>     container on specified host even though node has the resources.
>     I am using AMRMClient, only thing is I made following changes to
>     get the containers on the specified node.
>     /
>     //Line: 361 //
>     //Original//
>     // addResourceRequest(req.getPriority(), node,
>     req.getCapability(), req, true);//
>     //Modifiled//
>     // addResourceRequest(req.getPriority(), node,
>     req.getCapability(), req, req.getRelaxLocality());//
>     //
>     //Line 374//
>     //Original//
>     // addResourceRequest(req.getPriority(), rack,
>     req.getCapability(), req, true);//
>     //Modifiled//
>     // addResourceRequest(req.getPriority(), rack,
>     req.getCapability(), req, req.getRelaxLocality());//
>     ///
>
>
>     Thanks
>     -Gaurav
>
>
>     On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>
>         Yeah, specifying a host name with relaxLocality is meaningful.
>          Schedulers
>         use delay scheduling (
>         http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf
>         <http://www.cs.berkeley.edu/%7Ematei/talks/2010/eurosys_delaysched.pdf>)
>         to
>         achieve locality when relaxLocality is on.  But it is turned
>         off by
>         default.  The individual scheduler docs have specifics on how
>         to configure
>         it.
>
>         Guarav,
>         Using ResourceRequests directly is not straightforward and
>         error prone.  Is
>         there a reason that AMRMClient is unsuitable for your needs?
>
>         -Sandy
>
>
>
>         On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise
>         <thomas.weise@gmail.com <mailto:thomas.weise@gmail.com>>wrote:
>
>             Is it possible to specify a particular node and have RM
>             fallback to an
>             different node only after making an attempt to allocate
>             for the requested
>             node? In other words, is the combination of specific host
>             name and
>             relaxLocality=TRUE meaningful at all?
>
>             Thanks.
>
>
>             On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur
>             <tucu@cloudera.com <mailto:tucu@cloudera.com>
>
>                 wrote:
>                 Gaurav,
>
>                 Setting relaxLocality to FALSE should do it.
>
>                 thanks.
>
>
>                 On Wed, Nov 13, 2013 at 2:58 PM, gaurav
>                 <gaurav@datatorrent.com
>                 <mailto:gaurav@datatorrent.com>> wrote:
>
>                       Hi,
>                       I am trying to allocate containers on a
>                     particular node in Yarn but
>
>             Yarn
>
>                     is returning me containers on different node
>                     although the requested
>
>             node
>
>                     has resources available.
>
>                     I checked into the allocate(AllocateRequest
>                     request) function of
>                     ApplicationMasterService and my request is as follows
>
>                     *request: ask { priority { priority: 1 }
>                     resource_name: "h2"
>
>             capability {
>
>                     memory: 1000 } num_containers: 2 } ask { priority
>                     { priority: 1 }
>                     resource_name: "/default-rack" capability {
>                     memory: 1000 }
>
>                 num_containers:
>
>                     2 } ask { priority { priority: 1 } resource_name:
>                     "*" capability {
>
>                 memory:
>
>                     1000 } num_containers: 2 } response_id: 1
>                     progress: 0.0*
>
>                     but the containers that I am getting back is as
>                     follows
>                     [Container: [ContainerId:
>                     container_1384381084244_0001_01_000002,
>
>             NodeId:
>
>                     h1:1234, NodeHttpAddress: h1:2, Resource:
>                     <memory:1024, vCores:1>,
>                     Priority: 1, Token: Token { kind: ContainerToken,
>                     service: h1:1234 },
>
>             ],
>
>                     Container: [ContainerId:
>                     container_1384381084244_0001_01_000003,
>
>             NodeId:
>
>                     h1:1234, NodeHttpAddress: h1:2, Resource:
>                     <memory:1024, vCores:1>,
>                     Priority: 1, Token: Token { kind: ContainerToken,
>                     service: h1:1234 },
>
>             ]]
>
>                     I am attaching the test case that I have written
>                     along with the mail.
>
>             It
>
>                     uses classes under
>                     org.apache.hadoop.yarn.server.resourcemanager
>
>             package.
>
>                     Any pointers would be of great help
>
>                     Thanks
>                     Gaurav
>
>
>
>
>
>
>                 --
>                 Alejandro
>
>
>


Mime
View raw message