Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (nike.apache.org: domain of tim@mesosphere.io designates
 209.85.223.197 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAC2U_PdqGm6TJhBhM_tLb+gW2m7H22wH66qd2gM84OxML+7rTQ@mail.gmail.com>
References: <1420436255756-20966.post@n3.nabble.com>
	<CAC2U_PdqGm6TJhBhM_tLb+gW2m7H22wH66qd2gM84OxML+7rTQ@mail.gmail.com>
Date: Mon, 5 Jan 2015 17:07:25 -0800
Message-ID: 
 <CAC2U_PfysXsNEqwN9+WnNfioLRds=FXU9pw6UwAHB+kvZAt=vg@mail.gmail.com>
Subject: Fwd: Controlling number of executors on Mesos vs YARN
From: Tim Chen <tim@mesosphere.io>
To: user <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=047d7b471e3a1308fe050bf16e37

--047d7b471e3a1308fe050bf16e37
Content-Type: text/plain; charset=UTF-8

Forgot to hit reply-all.

---------- Forwarded message ----------
From: Tim Chen <tim@mesosphere.io>
Date: Sun, Jan 4, 2015 at 10:46 PM
Subject: Re: Controlling number of executors on Mesos vs YARN
To: mvle <mvle@us.ibm.com>


Hi Mike,

You're correct there is no such setting in for Mesos coarse grain mode,
since the assumption is that each node is launched with one container and
Spark is launching multiple tasks in that container.

In fine-grain mode there isn't a setting like that, as it currently will
launch an executor as long as it satisfies the minimum container resource
requirement.

I've created a JIRA earlier about capping the number of executors or better
distribute the # of executors launched in each node. Since the decision of
choosing what node to launch containers is all in the Spark scheduler side,
it's very easy to modify it.

Btw, what's the configuration to set the # of executors on YARN side?

Thanks,

Tim


On Sun, Jan 4, 2015 at 9:37 PM, mvle <mvle@us.ibm.com> wrote:

> I'm trying to compare the performance of Spark running on Mesos vs YARN.
> However, I am having problems being able to configure the Spark workload to
> run in a similar way on Mesos and YARN.
>
> When running Spark on YARN, you can specify the number of executors per
> node. So if I have a node with 4 CPUs, I can specify 6 executors on that
> node. When running Spark on Mesos, there doesn't seem to be an equivalent
> way to specify this. In Mesos, you can somewhat force this by specifying
> the
> number of CPU resources to be 6 when running the slave daemon. However,
> this
> seems to be a static configuration of the Mesos cluster rather something
> that can be configured in the Spark framework.
>
> So here is my question:
>
> For Spark on Mesos, am I correct that there is no way to control the number
> of executors per node (assuming an idle cluster)? For Spark on Mesos
> coarse-grained mode, there is a way to specify max_cores but that is still
> not equivalent to specifying the number of executors per node as when Spark
> is run on YARN.
>
> If I am correct, then it seems Spark might be at a disadvantage running on
> Mesos compared to YARN (since it lacks the fine tuning ability provided by
> YARN).
>
> Thanks,
> Mike
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Controlling-number-of-executors-on-Mesos-vs-YARN-tp20966.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

--047d7b471e3a1308fe050bf16e37
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Forgot to hit reply-all.<div><br><div class=3D"gmail_quote=
">---------- Forwarded message ----------<br>From: <b class=3D"gmail_sender=
name">Tim Chen</b> <span dir=3D"ltr">&lt;<a href=3D"mailto:tim@mesosphere.i=
o">tim@mesosphere.io</a>&gt;</span><br>Date: Sun, Jan 4, 2015 at 10:46 PM<b=
r>Subject: Re: Controlling number of executors on Mesos vs YARN<br>To: mvle=
 &lt;<a href=3D"mailto:mvle@us.ibm.com">mvle@us.ibm.com</a>&gt;<br><br><br>=
<div dir=3D"ltr">Hi Mike,<div><br></div><div>You&#39;re correct there is no=
 such setting in for Mesos coarse grain mode, since the assumption is that =
each node is launched with one container and Spark is launching multiple ta=
sks in that container.</div><div><br></div><div>In fine-grain mode there is=
n&#39;t a setting like that, as it currently will launch an executor as lon=
g as it satisfies the minimum container resource requirement.</div><div><br=
></div><div>I&#39;ve created a JIRA earlier about capping the number of exe=
cutors or better distribute the # of executors launched in each node. Since=
 the decision of choosing what node to launch containers is all in the Spar=
k scheduler side, it&#39;s very easy to modify it.</div><div><br></div><div=
>Btw, what&#39;s the configuration to set the # of executors on YARN side?<=
/div><div><br></div><div>Thanks,</div><div><br></div><div>Tim</div><div><br=
></div><div><br></div></div><div class=3D"HOEnZb"><div class=3D"h5"><div cl=
ass=3D"gmail_extra"><br><div class=3D"gmail_quote">On Sun, Jan 4, 2015 at 9=
:37 PM, mvle <span dir=3D"ltr">&lt;<a href=3D"mailto:mvle@us.ibm.com" targe=
t=3D"_blank">mvle@us.ibm.com</a>&gt;</span> wrote:<br><blockquote class=3D"=
gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-=
left:1ex">I&#39;m trying to compare the performance of Spark running on Mes=
os vs YARN.<br>
However, I am having problems being able to configure the Spark workload to=
<br>
run in a similar way on Mesos and YARN.<br>
<br>
When running Spark on YARN, you can specify the number of executors per<br>
node. So if I have a node with 4 CPUs, I can specify 6 executors on that<br=
>
node. When running Spark on Mesos, there doesn&#39;t seem to be an equivale=
nt<br>
way to specify this. In Mesos, you can somewhat force this by specifying th=
e<br>
number of CPU resources to be 6 when running the slave daemon. However, thi=
s<br>
seems to be a static configuration of the Mesos cluster rather something<br=
>
that can be configured in the Spark framework.<br>
<br>
So here is my question:<br>
<br>
For Spark on Mesos, am I correct that there is no way to control the number=
<br>
of executors per node (assuming an idle cluster)? For Spark on Mesos<br>
coarse-grained mode, there is a way to specify max_cores but that is still<=
br>
not equivalent to specifying the number of executors per node as when Spark=
<br>
is run on YARN.<br>
<br>
If I am correct, then it seems Spark might be at a disadvantage running on<=
br>
Mesos compared to YARN (since it lacks the fine tuning ability provided by<=
br>
YARN).<br>
<br>
Thanks,<br>
Mike<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href=3D"http://apache-spark-user-list.1001=
560.n3.nabble.com/Controlling-number-of-executors-on-Mesos-vs-YARN-tp20966.=
html" target=3D"_blank">http://apache-spark-user-list.1001560.n3.nabble.com=
/Controlling-number-of-executors-on-Mesos-vs-YARN-tp20966.html</a><br>
Sent from the Apache Spark User List mailing list archive at Nabble.com.<br=
>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href=3D"mailto:user-unsubscribe@spark.apache.org=
" target=3D"_blank">user-unsubscribe@spark.apache.org</a><br>
For additional commands, e-mail: <a href=3D"mailto:user-help@spark.apache.o=
rg" target=3D"_blank">user-help@spark.apache.org</a><br>
<br>
</blockquote></div><br></div>
</div></div></div><br></div></div>

--047d7b471e3a1308fe050bf16e37--