Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A20281852B for ; Tue, 15 Dec 2015 10:07:13 +0000 (UTC) Received: (qmail 95653 invoked by uid 500); 15 Dec 2015 10:07:11 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 95524 invoked by uid 500); 15 Dec 2015 10:07:11 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 95512 invoked by uid 99); 15 Dec 2015 10:07:11 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Dec 2015 10:07:11 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 632F71804C2 for ; Tue, 15 Dec 2015 10:07:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.901 X-Spam-Level: *** X-Spam-Status: No, score=3.901 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 514bjlUW7Muy for ; Tue, 15 Dec 2015 10:06:55 +0000 (UTC) Received: from mail-ig0-f175.google.com (mail-ig0-f175.google.com [209.85.213.175]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 49A21439B3 for ; Tue, 15 Dec 2015 10:06:55 +0000 (UTC) Received: by mail-ig0-f175.google.com with SMTP id ph11so104312185igc.1 for ; Tue, 15 Dec 2015 02:06:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=p01E0ypoWsZGemihIxsIN5Ufqe/m2ZdVcYM3qML7DqA=; b=cMtEMa51Nlj0mNvoynlUucGWMLPL96ZZfFTDLV1T1AKCXPyHC7HQCslNgSDEMSRndW Yl6IXj+N60mWfZWRa9gIZkMUJw+oCUOijV1qdkwdRPgXy2xlv1sm4C9xefS7H6oXTnHw 19psi7SHaQ4e2ua2ObnG0ZP8A2es9rP1KtvAP8pNf6W9dsVEB0TAMaHk26XZNt9cVkLV V7Ja9qlGtcAnZqeXyXnP4wU3uQpgoEbUXOGdndLDiHOR/tG37Nt+z0KdK0uuBkd1e7IW MxePKLO3sievoB0gfAsUjRUVq9j84UHWPhhp3LHN0jR+2aDDj1ncFy8wMJKs2tlh4ZhE D26Q== MIME-Version: 1.0 X-Received: by 10.50.56.110 with SMTP id z14mr3330481igp.0.1450174014921; Tue, 15 Dec 2015 02:06:54 -0800 (PST) Received: by 10.36.121.1 with HTTP; Tue, 15 Dec 2015 02:06:54 -0800 (PST) In-Reply-To: <006401d1371e$aa943040$ffbc90c0$@com> References: <005601d1371a$41d6d250$c58476f0$@com> <48EBA7FE-1BA3-405B-801F-56BD6A81E6E2@gmail.com> <006401d1371e$aa943040$ffbc90c0$@com> Date: Tue, 15 Dec 2015 18:06:54 +0800 Message-ID: Subject: Re: spark with label nodes in yarn From: Saisai Shao To: =?UTF-8?B?5byg5b+X5by6KOaXuui9qSk=?= Cc: Ted Yu , dev Content-Type: multipart/alternative; boundary=089e0158a9ce05c5e70526ecf314 --089e0158a9ce05c5e70526ecf314 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable SPARK-6470 only supports node label expression for executors. SPARK-7173 supports node label expression for AM (will be in 1.6). If you want to schedule your whole application through label expression, you have to configure both am and executor label expression. If you only want to schedule executors through label expression, the executor configuration is enough, but you have to make sure your cluster has some nodes with no label. You can refer to this document ( http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mg= t/content/configuring_node_labels.html ). Thanks Saisai On Tue, Dec 15, 2015 at 5:55 PM, =E5=BC=A0=E5=BF=97=E5=BC=BA(=E6=97=BA=E8= =BD=A9) wrote: > Hi Ted, > > > > Thanks for your quick response, but I think the link you gave it to me is > more advanced feature. > > Yes, I noticed SPARK-6470(https://issues.apache.org/jira/browse/SPARK-647= 0) > > > And I just tried for this feature with spark 1.5.0, what happened to me > was I was blocked to get the YARN containers by setting > spark.yarn.executor.nodeLabelExpression property. My question, > https://issues.apache.org/jira/browse/SPARK-7173 will fix this? > > > > Thanks > > Allen > > > > > > *=E5=8F=91=E4=BB=B6=E4=BA=BA:* Ted Yu [mailto:yuzhihong@gmail.com] > *=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4:* 2015=E5=B9=B412=E6=9C=8815=E6=97= =A5 17:39 > *=E6=94=B6=E4=BB=B6=E4=BA=BA:* =E5=BC=A0=E5=BF=97=E5=BC=BA(=E6=97=BA=E8= =BD=A9) > *=E6=8A=84=E9=80=81:* dev@spark.apache.org > *=E4=B8=BB=E9=A2=98:* Re: spark with label nodes in yarn > > > > Please take a look at: > > https://issues.apache.org/jira/browse/SPARK-7173 > > > > Cheers > > > On Dec 15, 2015, at 1:23 AM, =E5=BC=A0=E5=BF=97=E5=BC=BA(=E6=97=BA=E8=BD= =A9) wrote: > > Hi all, > > > > Has anyone tried label based scheduling via spark on yarn? I=E2=80=99ve t= ried > that, it didn=E2=80=99t work, spark 1.4.1 + apache hadoop 2.6.0 > > > > Any feedbacks are welcome. > > > > Thanks > > Allen > > --089e0158a9ce05c5e70526ecf314 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
SPARK-6470 only supports node label expression for executo= rs.
SPARK-7173 supports node label expression for AM (will be in 1.6).<= /div>

If you want to schedule your whole application thr= ough label expression, you have to configure both am and executor label exp= ression. If you only want to schedule executors through label expression, t= he executor configuration is enough, but you have to make sure your cluster= has some nodes with no label.


Thanks
Saisai


On Tue, Dec 15, 2015 at 5:55 PM, =E5=BC=A0=E5=BF=97=E5=BC= =BA(=E6=97=BA=E8=BD=A9) <zzq98736@alibaba-inc.com> wr= ote:

Hi Ted,=

= =C2=A0

Thanks = for your quick response, but I think the link you gave it to me is more advanced feature.<= u>

Yes, I = noticed SPARK-6470(https://issues.apache.org/jira/browse/SPARK-6470= )

And I j= ust tried for this feature with spark 1.5.0, what happened to me was I was blocked to get= the YARN containers by setting spark.yarn.executor.nodeLabelExpression property= . My question, https://issues.apache.org/jira/br= owse/SPARK-7173 will fix this?

= =C2=A0

Thanks<= u>

Allen

= =C2=A0

= =C2=A0

=E5=8F=91=E4=BB=B6= =E4=BA=BA: Ted Yu [mailto:yuzhihong@= gmail.com]
= =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2015=E5=B9=B412=E6=9C=8815=E6=97=A5 17:39
=E6=94=B6=E4=BB=B6=E4=BA=BA: =E5=BC=A0=E5=BF=97=E5=BC=BA(=E6=97=BA=E8=BD=A9)
=E6=8A=84=E9=80=81: dev@spark= .apache.org
=E4=B8=BB=E9=A2=98: Re: spark with label nodes in yarn

=C2=A0

Please take a look at:

=C2=A0

Cheers

=
On Dec 15, 2015, at 1:23 AM, =E5=BC=A0=E5=BF=97=E5=BC=BA(=E6=97=BA=E8=BD=A9) <zzq98736@= alibaba-inc.com> wrote:

Hi all,

=C2=A0

Has anyone tried label based sc= heduling via spark on yarn? I=E2=80=99ve tried that, it didn=E2=80=99t work, spark 1.4.1= + apache hadoop 2.6.0

=C2=A0

Any feedbacks are welcome.

=C2=A0

Thanks

Allen


--089e0158a9ce05c5e70526ecf314--