hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gaurav Gupta" <gau...@datatorrent.com>
Subject RE: Allocating Containers on a particular Node in Yarn
Date Thu, 14 Nov 2013 21:28:25 GMT
Sandy,

 

For the first question you mentioned relaxation locality is turn on, I
assume you mean that relax locality is set to false right?

 

For scheduler, I am using capacity scheduler and modified following property
in capacity-scheduler.xml

<property>

<name>yarn.scheduler.capacity.node-locality-delay</name>

     <value>1</value>

     <description>

       Number of missed scheduling opportunities after which the
CapacityScheduler

       attempts to schedule rack-local containers.

       Typically this should be set to number of racks in the cluster, this

       feature is disabled by default, set to -1.

     </description>

   </property>

 

-Gaurav

 

From: Sandy Ryza [mailto:sandy.ryza@cloudera.com] 
Sent: Thursday, November 14, 2013 12:41 PM
To: user@hadoop.apache.org
Subject: Re: Allocating Containers on a particular Node in Yarn

 

Great to hear.  Other answers inline

 

On Thu, Nov 14, 2013 at 12:05 PM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

Sandy,

 

The last trick worked but now I have couple of more questions

 

1.      If I don't request for rack and relax locality is false with
scheduler delay on , I see that when I pass a wrong host I don't get any
container back. Why so?

When locality relaxation is turned on, we will only ever place the container
on the host that you requested.  This means that if no such host exists, we
will never place the container.  This behavior could maybe be improved, but
there are some tricky issues to deal with there about nodes going down and
such. 

2.      I also noticed that with scheduler delay on and relax locality set
to false, If I request for rack along with a host I get one container on
expected node but not other containers. 

3.      With scheduler delay on and relax locality set to true without
requesting rack, I don't get the containers on the required host

What scheduler are you using and what properties are you using to turn the
scheduler delay on? 

 

Thanks

-Gaurav

 

From: Sandy Ryza [mailto:sandy.ryza@cloudera.com] 
Sent: Thursday, November 14, 2013 11:41 AM
To: user@hadoop.apache.org


Subject: Re: Allocating Containers on a particular Node in Yarn

 

Requesting the rack is not necessary, and is leading to the behavior that
you're seeing.

 

The documentation states:

   * <li>If locality relaxation is disabled, then only within the same
request,  

   * a node and its rack may be specified together. This allows for a
specific   

   * rack with a preference for a specific node within that rack.</li>

 

 

On Wed, Nov 13, 2013 at 10:10 PM, gaurav <gaurav@datatorrent.com> wrote:

Here is the snippet of code that I am using to allocate containers

    AMRMClient<ContainerRequest> amRmClient =
AMRMClient.createAMRMClient();;
    String host = "h1";
    Resource capability = Records.newRecord(Resource.class);
    capability.setMemory(memory);
    nodes = new String[] {host};
    // in order to request a host, we also have to request the rack
    racks = new String[] {"/default-rack"};
     List<ContainerRequest> containerRequests = new
ArrayList<ContainerRequest>();
    List<ContainerId> releasedContainers = new ArrayList<ContainerId>();
    containerRequests.add(new ContainerRequest(capability, nodes, racks,
Priority.newInstance(priority),false));
    if (containerRequests.size() > 0) {
      LOG.info("Asking RM for containers: " + containerRequests);
      for (ContainerRequest cr : containerRequests) {
        LOG.info("Requested container: {}", cr.toString());
        amRmClient.addContainerRequest(cr);
      }
    }

    for (ContainerId containerId : releasedContainers) {
      LOG.info("Released container, id={}", containerId.getId());
      amRmClient.releaseAssignedContainer(containerId);
    }
    return amRmClient.allocate(0);



-Gaurav

 

On 11/13/2013 07:36 PM, Sandy Ryza wrote:

In that case, the AMRMClient code looks correct to me.  Can you share the
code you've written against it that's not receiving the correct containers?

 

On Wed, Nov 13, 2013 at 5:30 PM, gaurav <gaurav@datatorrent.com> wrote:

Hi,

I looked in the trunk and line numbers are 361 and 366.

Thanks
-Gaurav

On 11/13/2013 5:04 PM, gaurav wrote:

I have hadoop-2.2.0

Thanks
-Gaurav

On 11/13/2013 4:59 PM, Sandy Ryza wrote:

What version are you using?  Setting the relax locality to true for nodes is
always correct.  For racks, this is not necessarily the case.  When I look
at trunk, it is set to true always on line 361 (which is correct), but on on
line 374. 

 

-Sandy

 

On Wed, Nov 13, 2013 at 4:47 PM, gaurav <gaurav@datatorrent.com> wrote:

Hi Sandy,

No it is not working for me.  As mentioned earlier, AMRMClient is not
respecting the locality for node and rack in Line 361 and 374 and is set to
true always.
I am requesting for one container with specified node and rack and relaxed
locality false.

Thanks
-Gaurav

On 11/13/2013 4:24 PM, Sandy Ryza wrote:

[moving to user list]

 

Right.  relaxLocality needs to be set on the next level up.  It determines
whether locality can be relaxed to that level.  Confusing, I know.  If you
are using AMRMClient, you should be able to accomplish what you're looking
for by creating an AMRMClient.ContainerRequest that specifies a node and
sets relax locality to false.  Is that not working for you? 

 

-Sandy

 

On Wed, Nov 13, 2013 at 4:19 PM, gaurav <gaurav@datatorrent.com> wrote:

Hi Sandy,

If I set relaxlocality to true with host name, I don't get the container on
specified host even though node has the resources.
I am using AMRMClient, only thing is I made following changes to get the
containers on the specified node.
/
//Line: 361 //
//Original//
// addResourceRequest(req.getPriority(), node, req.getCapability(), req,
true);//
//Modifiled//
// addResourceRequest(req.getPriority(), node, req.getCapability(), req,
req.getRelaxLocality());//
//
//Line 374//
//Original//
// addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
true);//
//Modifiled//
// addResourceRequest(req.getPriority(), rack, req.getCapability(), req,
req.getRelaxLocality());//
///


Thanks
-Gaurav 



On 11/13/2013 4:02 PM, Sandy Ryza wrote:

Yeah, specifying a host name with relaxLocality is meaningful.  Schedulers
use delay scheduling (
http://www.cs.berkeley.edu/~matei/talks/2010/eurosys_delaysched.pdf
<http://www.cs.berkeley.edu/%7Ematei/talks/2010/eurosys_delaysched.pdf> ) to
achieve locality when relaxLocality is on.  But it is turned off by
default.  The individual scheduler docs have specifics on how to configure
it.

Guarav,
Using ResourceRequests directly is not straightforward and error prone.  Is
there a reason that AMRMClient is unsuitable for your needs?

-Sandy



On Wed, Nov 13, 2013 at 3:55 PM, Thomas Weise <thomas.weise@gmail.com>wrote:

Is it possible to specify a particular node and have RM fallback to an
different node only after making an attempt to allocate for the requested
node? In other words, is the combination of specific host name and
relaxLocality=TRUE meaningful at all?

Thanks.


On Wed, Nov 13, 2013 at 3:23 PM, Alejandro Abdelnur <tucu@cloudera.com

wrote:
Gaurav,

Setting relaxLocality to FALSE should do it.

thanks.


On Wed, Nov 13, 2013 at 2:58 PM, gaurav <gaurav@datatorrent.com> wrote:

  Hi,
  I am trying to allocate containers on a particular node in Yarn but

Yarn

is returning me containers on different node although the requested

node

has resources available.

I checked into the allocate(AllocateRequest request) function of
ApplicationMasterService and my request is as follows

*request: ask { priority { priority: 1 } resource_name: "h2"

capability {

memory: 1000 } num_containers: 2 } ask { priority { priority: 1 }
resource_name: "/default-rack" capability { memory: 1000 }

num_containers:

2 } ask { priority { priority: 1 } resource_name: "*" capability {

memory:

1000 } num_containers: 2 } response_id: 1 progress: 0.0*

but the containers that I am getting back is as follows
[Container: [ContainerId: container_1384381084244_0001_01_000002,

NodeId:

h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 },

],

Container: [ContainerId: container_1384381084244_0001_01_000003,

NodeId:

h1:1234, NodeHttpAddress: h1:2, Resource: <memory:1024, vCores:1>,
Priority: 1, Token: Token { kind: ContainerToken, service: h1:1234 },

]]

I am attaching the test case that I have written along with the mail.

It

uses classes under org.apache.hadoop.yarn.server.resourcemanager

package.

Any pointers would be of great help

Thanks
Gaurav






--
Alejandro

 

 

 

 

 

 

 

 

 

 


Mime
View raw message