Mailing-List: contact dev-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
Date: Mon, 19 Dec 2016 23:45:31 +0100 (CET)
From: Mehdi Meziane <mehdi.meziane@ldmobile.net>
To: Michael Gummelt <mgummelt@mesosphere.io>
Cc: user@mesos.apache.org, dev@mesos.apache.org, User <user@spark.apache.org>,
	dev@spark.apache.org, Sumit Chawla <sumitkchawla@gmail.com>
Message-ID: <683244062.533951482187531504.JavaMail.root@zappa>
In-Reply-To: <320478199.533931482187422891.JavaMail.root@zappa>
MIME-Version: 1.0
Subject: Re: Mesos Spark Fine Grained Execution - CPU count
Content-Type: multipart/alternative;
	boundary="----=_Part_27470_307207350.1482187531502"
archived-at: Mon, 19 Dec 2016 22:54:26 -0000

------=_Part_27470_307207350.1482187531502
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

We will be interested by the results if you give a try to Dynamic allocatio=
n with mesos !=20


----- Mail Original -----=20
De: "Michael Gummelt" <mgummelt@mesosphere.io>=20
=C3=80: "Sumit Chawla" <sumitkchawla@gmail.com>=20
Cc: user@mesos.apache.org, dev@mesos.apache.org, "User" <user@spark.apache.=
org>, dev@spark.apache.org=20
Envoy=C3=A9: Lundi 19 D=C3=A9cembre 2016 22h42:55 GMT +01:00 Amsterdam / Be=
rlin / Berne / Rome / Stockholm / Vienne=20
Objet: Re: Mesos Spark Fine Grained Execution - CPU count=20


> Is this problem of idle executors sticking around solved in Dynamic Resou=
rce Allocation? Is there some timeout after which Idle executors can just s=
hutdown and cleanup its resources.=20

Yes, that's exactly what dynamic allocation does. But again I have no idea =
what the state of dynamic allocation + mesos is.=20


On Mon, Dec 19, 2016 at 1:32 PM, Chawla,Sumit < sumitkchawla@gmail.com > wr=
ote:=20


Great. Makes much better sense now. What will be reason to have spark.mesos=
.mesosExecutor. cores more than 1, as this number doesn't include the numbe=
r of cores for tasks.=20


So in my case it seems like 30 CPUs are allocated to executors. And there a=
re 48 tasks so 48 + 30 =3D 78 CPUs. And i am noticing this gap of 30 is mai=
ntained till the last task exits. This explains the gap. Thanks everyone. I=
 am still not sure how this number 30 is calculated. ( Is it dynamic based =
on current resources, or is it some configuration. I have 32 nodes in my cl=
uster).=20


Is this problem of idle executors sticking around solved in Dynamic Resourc=
e Allocation? Is there some timeout after which Idle executors can just shu=
tdown and cleanup its resources.=20


Regards=20
Sumit Chawla=20


On Mon, Dec 19, 2016 at 12:45 PM, Michael Gummelt < mgummelt@mesosphere.io =
> wrote:=20


> I should preassume that No of executors should be less than number of tas=
ks.=20

No. Each executor runs 0 or more tasks.=20

Each executor consumes 1 CPU, and each task running on that executor consum=
es another CPU. You can customize this via spark.mesos.mesosExecutor.cores =
( https://github.com/apache/spark/blob/v1.6.3/docs/running-on-mesos.md ) an=
d spark.task.cpus ( https://github.com/apache/spark/blob/v1.6.3/docs/config=
uration.md )=20


On Mon, Dec 19, 2016 at 12:09 PM, Chawla,Sumit < sumitkchawla@gmail.com > w=
rote:=20


Ah thanks. looks like i skipped reading this " Neither will executors termi=
nate when they=E2=80=99re idle."=20


So in my job scenario, I should preassume that No of executors should be le=
ss than number of tasks. Ideally one executor should execute 1 or more task=
s. But i am observing something strange instead. I start my job with 48 par=
titions for a spark job. In mesos ui i see that number of tasks is 48, but =
no. of CPUs is 78 which is way more than 48. Here i am assuming that 1 CPU =
is 1 executor. I am not specifying any configuration to set number of cores=
 per executor.=20


Regards=20
Sumit Chawla=20


On Mon, Dec 19, 2016 at 11:35 AM, Joris Van Remoortere < joris@mesosphere.i=
o > wrote:=20


That makes sense. From the documentation it looks like the executors are no=
t supposed to terminate:=20
http://spark.apache.org/docs/latest/running-on-mesos.html#fine-grained-depr=
ecated=20


Note that while Spark tasks in fine-grained will relinquish cores as they t=
erminate, they will not relinquish memory, as the JVM does not give memory =
back to the Operating System. Neither will executors terminate when they=E2=
=80=99re idle.=20


I suppose your task to executor CPU ratio is low enough that it looks like =
most of the resources are not being reclaimed. If your tasks were using sig=
nificantly more CPU the amortized cost of the idle executors would not be s=
uch a big deal.=20


=E2=80=94=20
Joris Van Remoortere=20
Mesosphere=20

On Mon, Dec 19, 2016 at 11:26 AM, Timothy Chen < tnachen@gmail.com > wrote:=
=20


Hi Chawla,=20

One possible reason is that Mesos fine grain mode also takes up cores=20
to run the executor per host, so if you have 20 agents running Fine=20
grained executor it will take up 20 cores while it's still running.=20

Tim=20

On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit < sumitkchawla@gmail.com > wr=
ote:=20


> Hi=20
>=20
> I am using Spark 1.6. I have one query about Fine Grained model in Spark.=
=20
> I have a simple Spark application which transforms A -> B. Its a single=
=20
> stage application. To begin the program, It starts with 48 partitions.=20
> When the program starts running, in mesos UI it shows 48 tasks and 48 CPU=
s=20
> allocated to job. Now as the tasks get done, the number of active tasks=
=20
> number starts decreasing. How ever, the number of CPUs does not decrease=
=20
> propotionally. When the job was about to finish, there was a single=20
> remaininig task, however CPU count was still 20.=20
>=20
> My questions, is why there is no one to one mapping between tasks and cpu=
s=20
> in Fine grained? How can these CPUs be released when the job is done, so=
=20
> that other jobs can start.=20
>=20
>=20
> Regards=20
> Sumit Chawla=20


--=20


Michael Gummelt=20
Software Engineer=20
Mesosphere=20


--=20


Michael Gummelt=20
Software Engineer=20
Mesosphere=20

------=_Part_27470_307207350.1482187531502--