Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4DA4E109B0 for ; Wed, 13 Nov 2013 13:51:07 +0000 (UTC) Received: (qmail 57524 invoked by uid 500); 13 Nov 2013 13:50:58 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 57249 invoked by uid 500); 13 Nov 2013 13:50:55 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 57241 invoked by uid 99); 13 Nov 2013 13:50:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Nov 2013 13:50:55 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of john.lilley@redpoint.net designates 206.225.164.223 as permitted sender) Received: from [206.225.164.223] (HELO hub021-nj-7.exch021.serverdata.net) (206.225.164.223) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Nov 2013 13:50:48 +0000 Received: from MBX021-E3-NJ-2.exch021.domain.local ([10.240.4.78]) by HUB021-NJ-7.exch021.domain.local ([10.240.4.114]) with mapi id 14.03.0158.001; Wed, 13 Nov 2013 05:50:26 -0800 From: John Lilley To: "user@hadoop.apache.org" Subject: RE: worker affinity and YARN scheduling Thread-Topic: worker affinity and YARN scheduling Thread-Index: Ac7e6BLJ8SgMPbo2QUKR65Fz8pfSwAAqWTOAADl04SA= Date: Wed, 13 Nov 2013 13:50:25 +0000 Message-ID: <869970D71E26D7498BDAC4E1CA92226B86DBB120@MBX021-E3-NJ-2.exch021.domain.local> References: <869970D71E26D7498BDAC4E1CA92226B86DB8926@MBX021-E3-NJ-2.exch021.domain.local> <56251376-B3F5-46D7-8CD2-D8C1F12BFF46@hortonworks.com> In-Reply-To: <56251376-B3F5-46D7-8CD2-D8C1F12BFF46@hortonworks.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [173.160.43.60] Content-Type: multipart/alternative; boundary="_000_869970D71E26D7498BDAC4E1CA92226B86DBB120MBX021E3NJ2exch_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_869970D71E26D7498BDAC4E1CA92226B86DBB120MBX021E3NJ2exch_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks! john From: Arun C Murthy [mailto:acm@hortonworks.com] Sent: Monday, November 11, 2013 7:25 PM To: user@hadoop.apache.org Subject: Re: worker affinity and YARN scheduling On Nov 11, 2013, at 6:28 AM, John Lilley > wrote: I would like to better understand YARN's scheduling with named workers and = relaxedLocality=3D=3Dtrue. For example, suppose that I have a three-node c= luster with nodes A,B,C. Each node has capacity to run two tasks of the ki= nd I desire simultaneously. My AM then requests nine containers with worke= r-name set so that I am requesting three containers per worker. The cluste= r starts idle and has no other users. My questions: * Is it optimal to issue three ResourceRequests, each with numConta= iners=3D=3D3? (As opposed to nine requests) Correct, that is why the resource protocol is designed as it is i.e. reduce= #requests required. * Initially, I expect the RM to allocate two containers per node, a= nd I expect to have the containers match the named workers. Is this always= the case? Generally - yes. It does depend on the scheduler implementation though. * If the first task completes on worker "B", can I rely on the Reso= urceRequest for "B" to be fulfilled next? Generally - yes. * What techniques should be used to get the containers on the worke= rs I expect most often? Nothing special, you could use relaxLocality =3D false if you really want i= t on a specific node/rack. * What techniques should be used to reduce container allocation lat= ency, if possible? Typically, latency is very small - there are some ongoing enhancements to m= ake it better. hth, Arun Thanks John -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to= which it is addressed and may contain information that is confidential, pr= ivileged and exempt from disclosure under applicable law. If the reader of = this message is not the intended recipient, you are hereby notified that an= y printing, copying, dissemination, distribution, disclosure or forwarding = of this communication is strictly prohibited. If you have received this com= munication in error, please contact the sender immediately and delete it fr= om your system. Thank You. --_000_869970D71E26D7498BDAC4E1CA92226B86DBB120MBX021E3NJ2exch_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks!=

john

 <= /p>

From: Arun C M= urthy [mailto:acm@hortonworks.com]
Sent: Monday, November 11, 2013 7:25 PM
To: user@hadoop.apache.org
Subject: Re: worker affinity and YARN scheduling

 

 

On Nov 11, 2013, at 6:28 AM, John Lilley <john.lilley@redpoint.net> wrote= :



I would like to better understand YARN&= #8217;s scheduling with named workers and relaxedLocality=3D=3Dtrue.  = For example, suppose that I have a three-node cluster with nodes A,B,C.&nbs= p; Each node has capacity to run two tasks of the kind I desire simultaneousl= y.  My AM then requests nine containers with worker-name set so that I= am requesting three containers per worker.  The cluster starts idle a= nd has no other users.  My questions:

·         Is it optimal to issue three ResourceRequests, each with numContainers=3D=3D3= ?   (As opposed to nine requests)

 

Correct, that is why the resource protocol is design= ed as it is i.e. reduce #requests required.



·         Initially, I expect the RM to allocate two containers per node, and I expect to have = the containers match the named workers.  Is this always the case?=

 

Generally - yes. It does depend on the scheduler imp= lementation though.



·         If the first task completes on worker “B”, can I rely on the Reso= urceRequest for “B” to be fulfilled next?

 

Generally - yes.



·         What techniques should be used to get the containers on the workers I expect mo= st often?

 

Nothing special, you could use relaxLocality =3D fal= se if you really want it on a specific node/rack.



·         What techniques should be used to reduce container allocation latency, if possi= ble?

 

Typically, latency is very small - there are some on= going enhancements to make it better.

 

hth,

Arun



Thanks

John

 

 

--

Arun C. Murthy=

Hortonworks Inc.
http://hortonworks.com/<= /span>

 


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to= which it is addressed and may contain information that is confidential, pr= ivileged and exempt from disclosure under applicable law. If the reader of = this message is not the intended recipient, you are hereby notified that any printing, copying, disseminati= on, distribution, disclosure or forwarding of this communication is strictl= y prohibited. If you have received this communication in error, please cont= act the sender immediately and delete it from your system. Thank You.

--_000_869970D71E26D7498BDAC4E1CA92226B86DBB120MBX021E3NJ2exch_--