hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gaurav <gau...@datatorrent.com>
Subject Re: Allocating Containers on a particular Node in Yarn
Date Thu, 14 Nov 2013 06:10:28 GMT
Here is the snippet of code that I am using to allocate containers

     AMRMClient<ContainerRequest> amRmClient = 
AMRMClient.createAMRMClient();;
     String host = "h1";
     Resource capability = Records.newRecord(Resource.class);
     capability.setMemory(memory);
     nodes = new String[] {host};
     // in order to request a host, we also have to request the rack
     racks = new String[] {"/default-rack"};
      List<ContainerRequest> containerRequests = new 
ArrayList<ContainerRequest>();
     List<ContainerId> releasedContainers = new ArrayList<ContainerId>();
     containerRequests.add(new ContainerRequest(capability, nodes, 
racks, Priority.newInstance(priority),false));
     if (containerRequests.size() > 0) {
       LOG.info("Asking RM for containers: " + containerRequests);
       for (ContainerRequest cr : containerRequests) {
         LOG.info("Requested container: {}", cr.toString());
         amRmClient.addContainerRequest(cr);
       }
     }

     for (ContainerId containerId : releasedContainers) {
       LOG.info("Released container, id={}", containerId.getId());
       amRmClient.releaseAssignedContainer(containerId);
     }
     return amRmClient.allocate(0);



-Gaurav

On 11/13/2013 07:36 PM, Sandy Ryza wrote:
> In that case, the AMRMClient code looks correct to me.  Can you share 
> the code you've written against it that's not receiving the correct 
> containers?
>
>
> On Wed, Nov 13, 2013 at 5:30 PM, gaurav <gaurav@datatorrent.com 
> <mailto:gaurav@datatorrent.com>> wrote:
>
>     Hi,
>
>     I looked in the trunk and line numbers are 361 and 366.
>
>     Thanks
>     -Gaurav
>
>     On 11/13/2013 5:04 PM, gaurav wrote:
>>     I have hadoop-2.2.0
>>     Thanks
>>     -Gaurav
>>     On 11/13/2013 4:59 PM, Sandy Ryza wrote:
>>>     What version are you using?  Setting the relax locality to true
>>>     for nodes is always correct.  For racks, this is not necessarily
>>>     the case.  When I look at trunk, it is set to true always on
>>>     line 361 (which is correct), but on on line 374.
>>>
>>>     -Sandy
>>>
>>>
>>>     On Wed, Nov 13, 2013 at 4:47 PM, gaurav <gaurav@datatorrent.com
>>>     <mailto:gaurav@datatorrent.com>> wrote:
>>>
>>>         Hi Sandy,
>>>
>>>         No it is not working for me.  As mentioned earlier,
>>>         AMRMClient is not respecting the locality for node and rack
>>>         in *Line 361 and 374 *and is set to true always.
>>>         I am requesting for one container with specified node and
>>>         rack and relaxed locality false.
>>>
>>>         Thanks
>>>         -Gaurav
>>>
>>>         On 11/13/2013 4:24 PM, Sandy Ryza wrote:
>>>>         [moving to user list]
>>>>
>>>>         Right.  relaxLocality needs to be set on the next level up.
>>>>          It determines whether locality can be relaxed to that
>>>>         level.  Confusing, I know.  If you are using AMRMClient,
>>>>         you should be able to accomplish what you're looking for by
>>>>         creating an AMRMClient.ContainerRequest that specifies a
>>>>         node and sets relax locality to false.  Is that not working
>>>>         for you?
>>>>
>>>>         -Sandy
>>>>
>>>>
>>>>         On Wed, Nov 13, 2013 at 4:19 PM, gaurav
>>>>         <gaurav@datatorrent.com <mailto:gaurav@datatorrent.com>>
wrote:
>>>>
>>>>             Hi Sandy,
>>>>
>>>>             If I set relaxlocality to true with host name, I don't
>>>>             get the container on specified host even though node
>>>>             has the resources.
>>>>             I am using AMRMClient, only thing is I made following
>>>>             changes to get the containers on the specified node.
>>>>             /
>>>>             //Line: 361 //
>>>>             //Original//
>>>>             // addResourceRequest(req.getPriority(), node,
>>>>             req.getCapability(), req, true);//
>>>>             //Modifiled//
>>>>             // addResourceRequest(req.getPriority(), node,
>>>>             req.getCapability(), req, req.getRelaxLocality());//
>>>>             //
>>>>             //Line 374//
>>>>             //Original//
>>>>             // addResourceRequest(req.getPriority(), rack,
>>>>             req.getCapability(), req, true);//
>>>>             //Modifiled//
>>>>             // addResourceRequest(req.getPriority(), rack,
>>>>             req.getCapability(), req, req.getRelaxLocality());//
>>>>             ///
>>>>
>>>>
>>>>             Thanks
>>>>             -Gaurav
>>>>
>>>>
>>>>             On 11/13/2013 4:02 PM, Sandy Ryza wrote:
>>>>
>>>>                 Yeah, specifying a host name with relaxLocality is
>>>>                 meaningful.  Schedulers
>>>>                 use delay scheduling (
>>>>                 http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf
>>>>                 <http://www.cs.berkeley.edu/%7Ematei/talks/2010/eurosys_delaysched.pdf>)
>>>>                 to
>>>>                 achieve locality when relaxLocality is on.  But it
>>>>                 is turned off by
>>>>                 default.  The individual scheduler docs have
>>>>                 specifics on how to configure
>>>>                 it.
>>>>
>>>>                 Guarav,
>>>>                 Using ResourceRequests directly is not
>>>>                 straightforward and error prone.  Is
>>>>                 there a reason that AMRMClient is unsuitable for
>>>>                 your needs?
>>>>
>>>>                 -Sandy
>>>>
>>>>
>>>>
>>>>                 On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise
>>>>                 <thomas.weise@gmail.com
>>>>                 <mailto:thomas.weise@gmail.com>>wrote:
>>>>
>>>>                     Is it possible to specify a particular node and
>>>>                     have RM fallback to an
>>>>                     different node only after making an attempt to
>>>>                     allocate for the requested
>>>>                     node? In other words, is the combination of
>>>>                     specific host name and
>>>>                     relaxLocality=TRUE meaningful at all?
>>>>
>>>>                     Thanks.
>>>>
>>>>
>>>>                     On Wed, Nov 13, 2013 at 3:23 PM, Alejandro
>>>>                     Abdelnur <tucu@cloudera.com
>>>>                     <mailto:tucu@cloudera.com>
>>>>
>>>>                         wrote:
>>>>                         Gaurav,
>>>>
>>>>                         Setting relaxLocality to FALSE should do it.
>>>>
>>>>                         thanks.
>>>>
>>>>
>>>>                         On Wed, Nov 13, 2013 at 2:58 PM, gaurav
>>>>                         <gaurav@datatorrent.com
>>>>                         <mailto:gaurav@datatorrent.com>> wrote:
>>>>
>>>>                               Hi,
>>>>                               I am trying to allocate containers on
>>>>                             a particular node in Yarn but
>>>>
>>>>                     Yarn
>>>>
>>>>                             is returning me containers on different
>>>>                             node although the requested
>>>>
>>>>                     node
>>>>
>>>>                             has resources available.
>>>>
>>>>                             I checked into the
>>>>                             allocate(AllocateRequest request)
>>>>                             function of
>>>>                             ApplicationMasterService and my request
>>>>                             is as follows
>>>>
>>>>                             *request: ask { priority { priority: 1
>>>>                             } resource_name: "h2"
>>>>
>>>>                     capability {
>>>>
>>>>                             memory: 1000 } num_containers: 2 } ask
>>>>                             { priority { priority: 1 }
>>>>                             resource_name: "/default-rack"
>>>>                             capability { memory: 1000 }
>>>>
>>>>                         num_containers:
>>>>
>>>>                             2 } ask { priority { priority: 1 }
>>>>                             resource_name: "*" capability {
>>>>
>>>>                         memory:
>>>>
>>>>                             1000 } num_containers: 2 } response_id:
>>>>                             1 progress: 0.0*
>>>>
>>>>                             but the containers that I am getting
>>>>                             back is as follows
>>>>                             [Container: [ContainerId:
>>>>                             container_1384381084244_0001_01_000002,
>>>>
>>>>                     NodeId:
>>>>
>>>>                             h1:1234, NodeHttpAddress: h1:2,
>>>>                             Resource: <memory:1024, vCores:1>,
>>>>                             Priority: 1, Token: Token { kind:
>>>>                             ContainerToken, service: h1:1234 },
>>>>
>>>>                     ],
>>>>
>>>>                             Container: [ContainerId:
>>>>                             container_1384381084244_0001_01_000003,
>>>>
>>>>                     NodeId:
>>>>
>>>>                             h1:1234, NodeHttpAddress: h1:2,
>>>>                             Resource: <memory:1024, vCores:1>,
>>>>                             Priority: 1, Token: Token { kind:
>>>>                             ContainerToken, service: h1:1234 },
>>>>
>>>>                     ]]
>>>>
>>>>                             I am attaching the test case that I
>>>>                             have written along with the mail.
>>>>
>>>>                     It
>>>>
>>>>                             uses classes under
>>>>                             org.apache.hadoop.yarn.server.resourcemanager
>>>>
>>>>                     package.
>>>>
>>>>                             Any pointers would be of great help
>>>>
>>>>                             Thanks
>>>>                             Gaurav
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>                         --
>>>>                         Alejandro
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>


Mime
View raw message