spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: spark with label nodes in yarn
Date Tue, 15 Dec 2015 10:06:54 GMT
SPARK-6470 only supports node label expression for executors.
SPARK-7173 supports node label expression for AM (will be in 1.6).

If you want to schedule your whole application through label expression,
you have to configure both am and executor label expression. If you only
want to schedule executors through label expression, the executor
configuration is enough, but you have to make sure your cluster has some
nodes with no label.

You can refer to this document (
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_yarn_resource_mgt/content/configuring_node_labels.html
).

Thanks
Saisai


On Tue, Dec 15, 2015 at 5:55 PM, 张志强(旺轩) <zzq98736@alibaba-inc.com> wrote:

> Hi Ted,
>
>
>
> Thanks for your quick response, but I think the link you gave it to me is
> more advanced feature.
>
> Yes, I noticed SPARK-6470(https://issues.apache.org/jira/browse/SPARK-6470)
>
>
> And I just tried for this feature with spark 1.5.0, what happened to me
> was I was blocked to get the YARN containers by setting
> spark.yarn.executor.nodeLabelExpression property. My question,
> https://issues.apache.org/jira/browse/SPARK-7173 will fix this?
>
>
>
> Thanks
>
> Allen
>
>
>
>
>
> *发件人:* Ted Yu [mailto:yuzhihong@gmail.com]
> *发送时间:* 2015年12月15日 17:39
> *收件人:* 张志强(旺轩)
> *抄送:* dev@spark.apache.org
> *主题:* Re: spark with label nodes in yarn
>
>
>
> Please take a look at:
>
> https://issues.apache.org/jira/browse/SPARK-7173
>
>
>
> Cheers
>
>
> On Dec 15, 2015, at 1:23 AM, 张志强(旺轩) <zzq98736@alibaba-inc.com> wrote:
>
> Hi all,
>
>
>
> Has anyone tried label based scheduling via spark on yarn? I’ve tried
> that, it didn’t work, spark 1.4.1 + apache hadoop 2.6.0
>
>
>
> Any feedbacks are welcome.
>
>
>
> Thanks
>
> Allen
>
>

Mime
View raw message