Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 08E4910B1A for ; Wed, 7 Aug 2013 16:54:13 +0000 (UTC) Received: (qmail 93224 invoked by uid 500); 7 Aug 2013 16:54:06 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 92658 invoked by uid 500); 7 Aug 2013 16:54:05 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 92649 invoked by uid 99); 7 Aug 2013 16:54:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 16:54:03 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of acm@hortonworks.com designates 209.85.160.41 as permitted sender) Received: from [209.85.160.41] (HELO mail-pb0-f41.google.com) (209.85.160.41) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Aug 2013 16:53:57 +0000 Received: by mail-pb0-f41.google.com with SMTP id rp2so2137725pbb.28 for ; Wed, 07 Aug 2013 09:53:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=JJm2TBmcr3+my89xdcIv1HHsjh3Shj+AQmCHgeWBc7E=; b=i2mE5ufSIoNfvL2yNRgwi51ZYGKJmcRfGOChyv3rqZPUmA+99CL/A+T7mo4f/st+K6 tah9yNN/BNLXmzFTWZ4WT5xKSNZC0POVXreMPX/1CUXIme4d5PQV3L9XDBcHCQL09l3L 7XKtmPzsHTNoOBwxxcLhjZM4DsC+j4eR3gwB0T820Dl/ozTyInLf2eORswlWcmU9uLge lKKP6NnTYfmZ6L1bXxO0pEe/gm7Qs6gUrMWpTbRrZpZljZ2vajrQLzVbjIHdHzPM3C56 IuLrozTgjRDD7ERVl0y6WfcvzAznRRq17VecCSYzXOZQsQinAcmt532+grbFD9aK5ZC2 fRtA== X-Gm-Message-State: ALoCoQmowvNLBIo8l0fsjRdVHg9SfNk7Mzo3tPtcJJxjx9bihx7v+CsXBsKmJDTCt+f4xEnbOJZ+ X-Received: by 10.68.138.131 with SMTP id qq3mr1719542pbb.10.1375894415830; Wed, 07 Aug 2013 09:53:35 -0700 (PDT) Received: from [10.11.3.229] ([192.175.27.2]) by mx.google.com with ESMTPSA id ss8sm11439704pab.6.2013.08.07.09.53.33 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 07 Aug 2013 09:53:34 -0700 (PDT) From: Arun C Murthy Content-Type: multipart/alternative; boundary="Apple-Mail=_9F06472F-ED61-417D-B31D-814CE52D4566" Message-Id: <4194764D-2718-4264-B168-BDD14F1B0CBF@hortonworks.com> Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: whitelist feature of YARN Date: Wed, 7 Aug 2013 09:53:34 -0700 References: To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1508) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_9F06472F-ED61-417D-B31D-814CE52D4566 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Go ahead - it's very close, I'll probably create an rc2 in the next = couple of days. On Aug 7, 2013, at 2:06 AM, Krishna Kishore Bonagiri = wrote: > Sandy,=20 > Thanks again. I found RC1 for 2.1.0-beta available at = http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/ > Would this have the fix for YARN-521? and, can I use that? >=20 > -Kishore >=20 >=20 > On Wed, Aug 7, 2013 at 12:35 PM, Sandy Ryza = wrote: > Responses inline: >=20 >=20 > On Tue, Aug 6, 2013 at 11:55 PM, Krishna Kishore Bonagiri = wrote: > Hi Sandy, >=20 > Thanks for the reply and it is good to know YARN-521 is done! Please = answer my following questions=20 >=20 > 1) when is 2.1.0-beta going to be released? is it soon or do you = suggest me take it from the trunk or is there a recent release candidate = available? >=20 > We're very close and my guess would be no later than the end of the = month (don't hold me to this). > =20 > 2) I have recently changed my application to use the new Asynchronous = interfaces. I am hoping it works with that too, correct me if I am = wrong. > ContainerRequest is shared by the async interfaces as well so it = should work here.=20 >=20 > 3) Change in interface: >=20 > The old interface for ContainerRequest constructor used to be this: >=20 > public ContainerRequest(Resource capability, String[] nodes, > String[] racks, Priority priority, int containerCount); >=20 > where as now it is changed to=20 >=20 > a) public ContainerRequest(Resource capability, String[] nodes, > String[] racks, Priority priority) > & >=20 > b) public ContainerRequest(Resource capability, String[] nodes, > String[] racks, Priority priority, boolean relaxLocality) >=20 > that means the old argument containerCount is gone! How would I be = able to specify how many containers do I need? >=20 > We now expect that you submit a ContainerRequest for each container = you want. > =20 > -Kishore >=20 >=20 >=20 >=20 > On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza = wrote: > YARN-521, which brings whitelisting to the AMRMClient APIs, is now = included in 2.1.0-beta. Check out the doc for the relaxLocality = paramater in ContainerRequest in AMRMClient: = https://github.com/apache/hadoop-common/blob/trunk/hadoop-yarn-project/had= oop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/ap= i/AMRMClient.java and I can help clarify here if anything's confusing. >=20 > -Sandy >=20 >=20 > On Tue, Jul 9, 2013 at 2:54 AM, Krishna Kishore Bonagiri = wrote: > Hi Sandy, >=20 > Yes, I have been using AMRMClient APIs. I am planning to shift to = whatever way is this white list feature is supported with. But am not = sure what is meant by submitting ResourceRequests directly to RM. Can = you please elaborate on this or give me a pointer to some example code = on how to do it... >=20 > Thanks for the reply, >=20 > -Kishore >=20 >=20 > On Mon, Jul 8, 2013 at 10:53 PM, Sandy Ryza = wrote: > Hi Krishna, >=20 > =46rom your previous email, it looks like you are using the AMRMClient = APIs. Support for whitelisting is not yet supported through them. I am = working on this in YARN-521, which should be included in the next = release after 2.1.0-beta. If you are submitting ResourceRequests = directly to the RM, you can whitelist a node by > * setting the relaxLocality flag on the node-level ResourceRequest to = true > * setting the relaxLocality flag on the corresponding rack-level = ResourceRequest to false > * setting the relaxLocality flag on the corresponding any-level = ResourceRequest to false >=20 > -Sandy >=20 >=20 > On Mon, Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri = wrote: > Hi, >=20 > Can someone please point to some example code of how to use the = whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta = and want to use this feature.=20 >=20 > It would be great if you can point me to some description of what = this white listing feature is, I have gone through some JIRA logs = related to this but more concrete explanation would be helpful. >=20 > Thanks, > Kishore >=20 >=20 >=20 >=20 >=20 >=20 -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ --Apple-Mail=_9F06472F-ED61-417D-B31D-814CE52D4566 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 Go = ahead - it's very close, I'll probably create an rc2 in the next couple = of days.

On Aug 7, 2013, at 2:06 AM, Krishna Kishore = Bonagiri <write2kishore@gmail.com> = wrote:

Sandy, 
  Thanks again. I = found RC1 for 2.1.0-beta available at http://= people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc1/
   Would this have the fix for YARN-521? and, can I use = that?

-Kishore


On Wed, Aug 7, = 2013 at 12:35 PM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
Responses inline:


On Tue, Aug 6, 2013 at 11:55 PM, = Krishna Kishore Bonagiri <write2kishore@gmail.com> wrote:
Hi Sandy,

  Thanks for the reply and it is good to know YARN-521 is done! = Please answer my following questions 

1) when is 2.1.0-beta going to be released? is it = soon or do you suggest me take it from the trunk or is there a recent = release candidate available?

We're very = close and my guess would be no later than the end of the = month (don't hold me to this).
 
2) I have recently changed my = application to use the new Asynchronous interfaces. I am hoping it works = with that too, correct me if I am wrong.
ContainerRequest is shared by the async interfaces as well so = it should work here. 

3) Change in interface:

The old interface for ContainerRequest constructor = used to be this:

 public = ContainerRequest(Resource capability, String[] nodes,
  =       String[] racks, Priority priority, int = containerCount);

where as now it is changed = to 

a) public ContainerRequest(Resource = capability, String[] nodes,
        = String[] racks, Priority priority)
&

b) public ContainerRequest(Resource capability, String[] = nodes,
        String[] racks, Priority = priority, boolean relaxLocality)

that means the = old argument containerCount is gone! How would I be able to specify how = many containers do I need?

We now expect = that you submit a ContainerRequest for each container you = want.
 
-Kishore



On Wed, Aug 7, 2013 at 11:37 AM, Sandy Ryza = <sandy.ryza@cloudera.com> wrote:
YARN-521, which brings = whitelisting to the AMRMClient APIs, is now included in 2.1.0-beta. =  Check out the doc for the relaxLocality paramater in = ContainerRequest in AMRMClient: https://github.com/apache/hadoop-common/blob/trunk/hadoo= p-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/had= oop/yarn/client/api/AMRMClient.java and I can help clarify here = if anything's confusing.

-Sandy


On Tue, Jul 9, = 2013 at 2:54 AM, Krishna Kishore Bonagiri <write2kishore@gmail.com> wrote:
Hi = Sandy,

  Yes, I have been using AMRMClient APIs. I am = planning to shift to whatever way is this white list feature is = supported with. But am not sure what is meant by submitting = ResourceRequests directly to RM. Can you please elaborate on this or = give me a pointer to some example code on how to do it...

   Thanks for the = reply,

-Kishore


On Mon, Jul 8, = 2013 at 10:53 PM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
Hi = Krishna,

=46rom your previous email, it looks like = you are using the AMRMClient APIs.  Support for whitelisting is not = yet supported through them.  I am working on this in YARN-521, = which should be included in the next release after 2.1.0-beta.  If = you are submitting ResourceRequests directly to the RM, you can = whitelist a node by
* setting the relaxLocality flag on the node-level ResourceRequest = to true
* setting the relaxLocality flag on the = corresponding rack-level ResourceRequest to false
* = setting the relaxLocality flag on the corresponding any-level = ResourceRequest to false

-Sandy


On Mon, = Jul 8, 2013 at 6:48 AM, Krishna Kishore Bonagiri <write2kishore@gmail.com> wrote:
Hi,

  = Can someone please point to some example code of how to use the = whitelist feature of YARN, I have recently got RC1 for hadoop-2.1.0-beta = and want to use this feature. 

  It would be great if you can point me to some = description of what this white listing feature is, I have gone through = some JIRA logs related to this but more concrete explanation would be = helpful.

Thanks,
Kishore







--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

=

= --Apple-Mail=_9F06472F-ED61-417D-B31D-814CE52D4566--