Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9378D200BE9 for ; Mon, 26 Dec 2016 22:04:20 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 92100160B3B; Mon, 26 Dec 2016 21:04:20 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 41CB5160B0E for ; Mon, 26 Dec 2016 22:04:19 +0100 (CET) Received: (qmail 96088 invoked by uid 500); 26 Dec 2016 21:04:17 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 96049 invoked by uid 99); 26 Dec 2016 21:04:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Dec 2016 21:04:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id BB1821A088C for ; Mon, 26 Dec 2016 21:04:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.679 X-Spam-Level: * X-Spam-Status: No, score=1.679 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=mesosphere.io Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 1i1WokY8wAtm for ; Mon, 26 Dec 2016 21:04:13 +0000 (UTC) Received: from mail-ua0-f200.google.com (mail-ua0-f200.google.com [209.85.217.200]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id A893D5FB54 for ; Mon, 26 Dec 2016 21:04:12 +0000 (UTC) Received: by mail-ua0-f200.google.com with SMTP id d5so99862721uag.5 for ; Mon, 26 Dec 2016 13:04:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mesosphere.io; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=AtR3eo5+860FlNnAchV0iSKejPvZc0kEUQ++F/zvkzo=; b=VpBqpCGrOnrQg0E/yBgl74yy/74Mrc464feyCXKPjuHfIpqQUtovWQrRHEK4sxYaF2 vzYzBTYDtB7MXR3TAQtc+WZnHvCEMv5yleNYUZ7+RF8rdaFiQZFgGhmRwHRQuO/6395Z PulcersRYy3VUwkOzAEqUue/iE8YzIcCH5Y0Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=AtR3eo5+860FlNnAchV0iSKejPvZc0kEUQ++F/zvkzo=; b=QZjrfOPYUSQLjA+Z6HBovTfkjp3cXrlqQZNUu4CNEs0jS0CjnIuZE6DAg6Zpgovm8j VgwBB8ZfG9Cq3nsedBpXGSM9hKEbQFksxdfAbd6eYVPEhKaicp9K+cHeK28gP4qYlGTx D48sM/76QpDYoVor2HnVXRlQa+5baT3D6Thb0Qs1jSQ9uE1NeTnpom1sGNY5/zigGPkO 9YF49ZmHJ7iw9Xbrvjcw3gb91D3iJ57mZ7TKRNR0HTXXSvf79RiWsCOjwytaqG5sYN2P ER4K2Ezx/4uzphC/X5yUQiKRNXtnNliBvEwH9z/UIGkJRx1Yiib2ETBDqvYzIiuPTztO l0Bw== X-Gm-Message-State: AIkVDXKDC6OH9YVsjKU7+PPYRTj2yXkoq5bqFUa0IPHo5Mtw4TVJHvKOTb659wzlyN1RmCP4x7xzoHavzwuuwUzf X-Received: by 10.159.39.233 with SMTP id b96mr17838623uab.86.1482786245992; Mon, 26 Dec 2016 13:04:05 -0800 (PST) MIME-Version: 1.0 Received: by 10.103.122.130 with HTTP; Mon, 26 Dec 2016 13:04:05 -0800 (PST) In-Reply-To: References: <320478199.533931482187422891.JavaMail.root@zappa> <683244062.533951482187531504.JavaMail.root@zappa> From: Michael Gummelt Date: Mon, 26 Dec 2016 13:04:05 -0800 Message-ID: Subject: Re: Mesos Spark Fine Grained Execution - CPU count To: Jacek Laskowski Cc: Davies Liu , Dev , Timothy Chen , Mehdi Meziane , "user@mesos.apache.org" , User , "dev@spark.apache.org" Content-Type: multipart/alternative; boundary=94eb2c1230c4787f7d05449613f2 archived-at: Mon, 26 Dec 2016 21:04:20 -0000 --94eb2c1230c4787f7d05449613f2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable In fine-grained mode (which is deprecated), Spark tasks (which are threads) were implemented as Mesos tasks. When a Mesos task starts and stops, its underlying cgroup, and therefore the resources its consuming on the cluster, grows or shrinks based on the resources allocated to the tasks, which in Spark is just CPU. This is what I mean by CPU usage "elastically growing". However, all Mesos tasks are run by an "executor", which has its own resource allocation. In Spark, the executor is the JVM, and all memory is allocated to the executor, because JVMs can't relinquish memory. If memory were allocated to the tasks, then the cgroup's memory allocation would shrink when the task terminated, but the JVM's memory consumption would stay constant, and the JVM would OOM. And, without dynamic allocation, executors never terminate during the duration of a Spark job, because even if they're idle (no tasks), they still may be hosting shuffle files. That's why dynamic allocation depends on an external shuffle service. Since executors never terminate, and all memory is allocated to the executors, Spark jobs even in fine-grained mode only grow in memory allocation, they don't shrink. On Mon, Dec 26, 2016 at 12:39 PM, Jacek Laskowski wrote: > Hi Michael, > > That caught my attention... > > Could you please elaborate on "elastically grow and shrink CPU usage" > and how it really works under the covers? It seems that CPU usage is > just a "label" for an executor on Mesos. Where's this in the code? > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Mon, Dec 26, 2016 at 6:25 PM, Michael Gummelt > wrote: > >> Using 0 for spark.mesos.mesosExecutor.cores is better than dynamic > >> allocation > > > > Maybe for CPU, but definitely not for memory. Executors never shut dow= n > in > > fine-grained mode, which means you only elastically grow and shrink CPU > > usage, not memory. > > > > On Sat, Dec 24, 2016 at 10:14 PM, Davies Liu > wrote: > >> > >> Using 0 for spark.mesos.mesosExecutor.cores is better than dynamic > >> allocation, but have to pay a little more overhead for launching a > >> task, which should be OK if the task is not trivial. > >> > >> Since the direct result (up to 1M by default) will also go through > >> mesos, it's better to tune it lower, otherwise mesos could become the > >> bottleneck. > >> > >> spark.task.maxDirectResultSize > >> > >> On Mon, Dec 19, 2016 at 3:23 PM, Chawla,Sumit > >> wrote: > >> > Tim, > >> > > >> > We will try to run the application in coarse grain mode, and share t= he > >> > findings with you. > >> > > >> > Regards > >> > Sumit Chawla > >> > > >> > > >> > On Mon, Dec 19, 2016 at 3:11 PM, Timothy Chen > wrote: > >> > > >> >> Dynamic allocation works with Coarse grain mode only, we wasn't awa= re > >> >> a need for Fine grain mode after we enabled dynamic allocation > support > >> >> on the coarse grain mode. > >> >> > >> >> What's the reason you're running fine grain mode instead of coarse > >> >> grain + dynamic allocation? > >> >> > >> >> Tim > >> >> > >> >> On Mon, Dec 19, 2016 at 2:45 PM, Mehdi Meziane > >> >> wrote: > >> >> > We will be interested by the results if you give a try to Dynamic > >> >> allocation > >> >> > with mesos ! > >> >> > > >> >> > > >> >> > ----- Mail Original ----- > >> >> > De: "Michael Gummelt" > >> >> > =C3=80: "Sumit Chawla" > >> >> > Cc: user@mesos.apache.org, dev@mesos.apache.org, "User" > >> >> > , dev@spark.apache.org > >> >> > Envoy=C3=A9: Lundi 19 D=C3=A9cembre 2016 22h42:55 GMT +01:00 Amst= erdam / > Berlin > >> >> > / > >> >> > Berne / Rome / Stockholm / Vienne > >> >> > Objet: Re: Mesos Spark Fine Grained Execution - CPU count > >> >> > > >> >> > > >> >> >> Is this problem of idle executors sticking around solved in > Dynamic > >> >> >> Resource Allocation? Is there some timeout after which Idle > >> >> >> executors > >> >> can > >> >> >> just shutdown and cleanup its resources. > >> >> > > >> >> > Yes, that's exactly what dynamic allocation does. But again I ha= ve > >> >> > no > >> >> idea > >> >> > what the state of dynamic allocation + mesos is. > >> >> > > >> >> > On Mon, Dec 19, 2016 at 1:32 PM, Chawla,Sumit > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> Great. Makes much better sense now. What will be reason to hav= e > >> >> >> spark.mesos.mesosExecutor.cores more than 1, as this number > doesn't > >> >> include > >> >> >> the number of cores for tasks. > >> >> >> > >> >> >> So in my case it seems like 30 CPUs are allocated to executors. > And > >> >> there > >> >> >> are 48 tasks so 48 + 30 =3D 78 CPUs. And i am noticing this ga= p of > >> >> >> 30 is > >> >> >> maintained till the last task exits. This explains the gap. > >> >> >> Thanks > >> >> >> everyone. I am still not sure how this number 30 is calculated. > ( > >> >> >> Is > >> >> it > >> >> >> dynamic based on current resources, or is it some configuration. > I > >> >> have 32 > >> >> >> nodes in my cluster). > >> >> >> > >> >> >> Is this problem of idle executors sticking around solved in > Dynamic > >> >> >> Resource Allocation? Is there some timeout after which Idle > >> >> >> executors > >> >> can > >> >> >> just shutdown and cleanup its resources. > >> >> >> > >> >> >> > >> >> >> Regards > >> >> >> Sumit Chawla > >> >> >> > >> >> >> > >> >> >> On Mon, Dec 19, 2016 at 12:45 PM, Michael Gummelt < > >> >> mgummelt@mesosphere.io> > >> >> >> wrote: > >> >> >>> > >> >> >>> > I should preassume that No of executors should be less than > >> >> >>> > number > >> >> of > >> >> >>> > tasks. > >> >> >>> > >> >> >>> No. Each executor runs 0 or more tasks. > >> >> >>> > >> >> >>> Each executor consumes 1 CPU, and each task running on that > >> >> >>> executor > >> >> >>> consumes another CPU. You can customize this via > >> >> >>> spark.mesos.mesosExecutor.cores > >> >> >>> > >> >> >>> (https://github.com/apache/spark/blob/v1.6.3/docs/ > running-on-mesos.md) > >> >> and > >> >> >>> spark.task.cpus > >> >> >>> (https://github.com/apache/spark/blob/v1.6.3/docs/ > configuration.md) > >> >> >>> > >> >> >>> On Mon, Dec 19, 2016 at 12:09 PM, Chawla,Sumit > >> >> >>> >> >> > > >> >> >>> wrote: > >> >> >>>> > >> >> >>>> Ah thanks. looks like i skipped reading this "Neither will > >> >> >>>> executors > >> >> >>>> terminate when they=E2=80=99re idle." > >> >> >>>> > >> >> >>>> So in my job scenario, I should preassume that No of executor= s > >> >> >>>> should > >> >> >>>> be less than number of tasks. Ideally one executor should > execute > >> >> >>>> 1 > >> >> or more > >> >> >>>> tasks. But i am observing something strange instead. I start > my > >> >> >>>> job > >> >> with > >> >> >>>> 48 partitions for a spark job. In mesos ui i see that number o= f > >> >> >>>> tasks > >> >> is 48, > >> >> >>>> but no. of CPUs is 78 which is way more than 48. Here i am > >> >> >>>> assuming > >> >> that 1 > >> >> >>>> CPU is 1 executor. I am not specifying any configuration to > set > >> >> number of > >> >> >>>> cores per executor. > >> >> >>>> > >> >> >>>> Regards > >> >> >>>> Sumit Chawla > >> >> >>>> > >> >> >>>> > >> >> >>>> On Mon, Dec 19, 2016 at 11:35 AM, Joris Van Remoortere > >> >> >>>> wrote: > >> >> >>>>> > >> >> >>>>> That makes sense. From the documentation it looks like the > >> >> >>>>> executors > >> >> >>>>> are not supposed to terminate: > >> >> >>>>> > >> >> >>>>> http://spark.apache.org/docs/latest/running-on-mesos.html# > >> >> fine-grained-deprecated > >> >> >>>>>> > >> >> >>>>>> Note that while Spark tasks in fine-grained will relinquish > >> >> >>>>>> cores as > >> >> >>>>>> they terminate, they will not relinquish memory, as the JVM > does > >> >> not give > >> >> >>>>>> memory back to the Operating System. Neither will executors > >> >> terminate when > >> >> >>>>>> they=E2=80=99re idle. > >> >> >>>>> > >> >> >>>>> > >> >> >>>>> I suppose your task to executor CPU ratio is low enough that = it > >> >> >>>>> looks > >> >> >>>>> like most of the resources are not being reclaimed. If your > tasks > >> >> were using > >> >> >>>>> significantly more CPU the amortized cost of the idle executo= rs > >> >> would not be > >> >> >>>>> such a big deal. > >> >> >>>>> > >> >> >>>>> > >> >> >>>>> =E2=80=94 > >> >> >>>>> Joris Van Remoortere > >> >> >>>>> Mesosphere > >> >> >>>>> > >> >> >>>>> On Mon, Dec 19, 2016 at 11:26 AM, Timothy Chen > >> >> >>>>> > >> >> >>>>> wrote: > >> >> >>>>>> > >> >> >>>>>> Hi Chawla, > >> >> >>>>>> > >> >> >>>>>> One possible reason is that Mesos fine grain mode also takes > up > >> >> cores > >> >> >>>>>> to run the executor per host, so if you have 20 agents runni= ng > >> >> >>>>>> Fine > >> >> >>>>>> grained executor it will take up 20 cores while it's still > >> >> >>>>>> running. > >> >> >>>>>> > >> >> >>>>>> Tim > >> >> >>>>>> > >> >> >>>>>> On Fri, Dec 16, 2016 at 8:41 AM, Chawla,Sumit < > >> >> sumitkchawla@gmail.com> > >> >> >>>>>> wrote: > >> >> >>>>>> > Hi > >> >> >>>>>> > > >> >> >>>>>> > I am using Spark 1.6. I have one query about Fine Grained > >> >> >>>>>> > model in > >> >> >>>>>> > Spark. > >> >> >>>>>> > I have a simple Spark application which transforms A -> B. > >> >> >>>>>> > Its a > >> >> >>>>>> > single > >> >> >>>>>> > stage application. To begin the program, It starts with 4= 8 > >> >> >>>>>> > partitions. > >> >> >>>>>> > When the program starts running, in mesos UI it shows 48 > tasks > >> >> >>>>>> > and > >> >> >>>>>> > 48 CPUs > >> >> >>>>>> > allocated to job. Now as the tasks get done, the number o= f > >> >> >>>>>> > active > >> >> >>>>>> > tasks > >> >> >>>>>> > number starts decreasing. How ever, the number of CPUs do= es > >> >> >>>>>> > not > >> >> >>>>>> > decrease > >> >> >>>>>> > propotionally. When the job was about to finish, there wa= s > a > >> >> single > >> >> >>>>>> > remaininig task, however CPU count was still 20. > >> >> >>>>>> > > >> >> >>>>>> > My questions, is why there is no one to one mapping betwee= n > >> >> >>>>>> > tasks > >> >> >>>>>> > and cpus > >> >> >>>>>> > in Fine grained? How can these CPUs be released when the > job > >> >> >>>>>> > is > >> >> >>>>>> > done, so > >> >> >>>>>> > that other jobs can start. > >> >> >>>>>> > > >> >> >>>>>> > > >> >> >>>>>> > Regards > >> >> >>>>>> > Sumit Chawla > >> >> >>>>> > >> >> >>>>> > >> >> >>>> > >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> -- > >> >> >>> Michael Gummelt > >> >> >>> Software Engineer > >> >> >>> Mesosphere > >> >> >> > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Michael Gummelt > >> >> > Software Engineer > >> >> > Mesosphere > >> >> > >> > >> > >> > >> -- > >> - Davies > > > > > > > > > > -- > > Michael Gummelt > > Software Engineer > > Mesosphere > --=20 Michael Gummelt Software Engineer Mesosphere --94eb2c1230c4787f7d05449613f2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
In fine-grained mode (which is deprecated), Spar= k tasks (which are threads) were implemented as Mesos tasks.=C2=A0 When a M= esos task starts and stops, its underlying cgroup, and therefore the resour= ces its consuming on the cluster, grows or shrinks based on the resources a= llocated to the tasks, which in Spark is just CPU.=C2=A0 This is what I mea= n by CPU usage "elastically growing".

However, all M= esos tasks are run by an "executor", which has its own resource a= llocation.=C2=A0 In Spark, the executor is the JVM, and all memory is alloc= ated to the executor, because JVMs can't relinquish memory.=C2=A0 If me= mory were allocated to the tasks, then the cgroup's memory allocation w= ould shrink when the task terminated, but the JVM's memory consumption = would stay constant, and the JVM would OOM.

And, without dynam= ic allocation, executors never terminate during the duration of a Spark job= , because even if they're idle (no tasks), they still may be hosting sh= uffle files.=C2=A0 That's why dynamic allocation depends on an external= shuffle service.=C2=A0 Since executors never terminate, and all memory is = allocated to the executors, Spark jobs even in fine-grained mode only grow = in memory allocation, they don't shrink.

On Mon, Dec 26, 2016 at 12:39 PM, Jace= k Laskowski <jacek@japila.pl> wrote:
Hi Michael,

That caught my attention...

Could you please elaborate on "elastically grow and shrink CPU usage&q= uot;
and how it really works under the covers? It seems that CPU usage is
just a "label" for an executor on Mesos. Where's this in the = code?

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache= -spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Dec 26, 2016 at 6:25 PM, Michael Gummelt <mgummelt@mesosphere.io> wrote:
>> Using 0 for spark.mesos.mesosExecutor.cores is better than dy= namic
>> allocation
>
> Maybe for CPU, but definitely not for memory.=C2=A0 Executors never sh= ut down in
> fine-grained mode, which means you only elastically grow and shrink CP= U
> usage, not memory.
>
> On Sat, Dec 24, 2016 at 10:14 PM, Davies Liu <davies.liu@gmail.com> wrote:
>>
>> Using 0 for spark.mesos.mesosExecutor.cores is better than dy= namic
>> allocation, but have to pay a little more overhead for launching a=
>> task, which should be OK if the task is not trivial.
>>
>> Since the direct result (up to 1M by default) will also go through=
>> mesos, it's better to tune it lower, otherwise mesos could bec= ome the
>> bottleneck.
>>
>> spark.task.maxDirectResultSize
>>
>> On Mon, Dec 19, 2016 at 3:23 PM, Chawla,Sumit <sumitkchawla@gmail.com>
>> wrote:
>> > Tim,
>> >
>> > We will try to run the application in coarse grain mode, and = share the
>> > findings with you.
>> >
>> > Regards
>> > Sumit Chawla
>> >
>> >
>> > On Mon, Dec 19, 2016 at 3:11 PM, Timothy Chen <tnachen@gmail.com> wrote:
>> >
>> >> Dynamic allocation works with Coarse grain mode only, we = wasn't aware
>> >> a need for Fine grain mode after we enabled dynamic alloc= ation support
>> >> on the coarse grain mode.
>> >>
>> >> What's the reason you're running fine grain mode = instead of coarse
>> >> grain + dynamic allocation?
>> >>
>> >> Tim
>> >>
>> >> On Mon, Dec 19, 2016 at 2:45 PM, Mehdi Meziane
>> >> <mehdi.m= eziane@ldmobile.net> wrote:
>> >> > We will be interested by the results if you give a t= ry to Dynamic
>> >> allocation
>> >> > with mesos !
>> >> >
>> >> >
>> >> > ----- Mail Original -----
>> >> > De: "Michael Gummelt" <mgummelt@mesosphere.io>
>> >> > =C3=80: "Sumit Chawla" <sumitkchawla@gmail.com>
>> >> > Cc: user@me= sos.apache.org, dev@mesos.apach= e.org, "User"
>> >> > <user@sp= ark.apache.org>, dev@spark.a= pache.org
>> >> > Envoy=C3=A9: Lundi 19 D=C3=A9cembre 2016 22h42:55 GM= T +01:00 Amsterdam / Berlin
>> >> > /
>> >> > Berne / Rome / Stockholm / Vienne
>> >> > Objet: Re: Mesos Spark Fine Grained Execution - CPU = count
>> >> >
>> >> >
>> >> >> Is this problem of idle executors sticking aroun= d solved in Dynamic
>> >> >> Resource Allocation?=C2=A0 Is there some timeout= after which Idle
>> >> >> executors
>> >> can
>> >> >> just shutdown and cleanup its resources.
>> >> >
>> >> > Yes, that's exactly what dynamic allocation does= .=C2=A0 But again I have
>> >> > no
>> >> idea
>> >> > what the state of dynamic allocation + mesos is.
>> >> >
>> >> > On Mon, Dec 19, 2016 at 1:32 PM, Chawla,Sumit
>> >> > <sumitk= chawla@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Great.=C2=A0 Makes much better sense now.=C2=A0 = What will be reason to have
>> >> >> spark.mesos.mesosExecutor.cores more than 1= , as this number doesn't
>> >> include
>> >> >> the number of cores for tasks.
>> >> >>
>> >> >> So in my case it seems like 30 CPUs are allocate= d to executors.=C2=A0 And
>> >> there
>> >> >> are 48 tasks so 48 + 30 =3D=C2=A0 78 CPUs.=C2=A0= And i am noticing this gap of
>> >> >> 30 is
>> >> >> maintained till the last task exits.=C2=A0 This = explains the gap.
>> >> >> Thanks
>> >> >> everyone.=C2=A0 I am still not sure how this num= ber 30 is calculated.=C2=A0 (
>> >> >> Is
>> >> it
>> >> >> dynamic based on current resources, or is it som= e configuration.=C2=A0 I
>> >> have 32
>> >> >> nodes in my cluster).
>> >> >>
>> >> >> Is this problem of idle executors sticking aroun= d solved in Dynamic
>> >> >> Resource Allocation?=C2=A0 Is there some timeout= after which Idle
>> >> >> executors
>> >> can
>> >> >> just shutdown and cleanup its resources.
>> >> >>
>> >> >>
>> >> >> Regards
>> >> >> Sumit Chawla
>> >> >>
>> >> >>
>> >> >> On Mon, Dec 19, 2016 at 12:45 PM, Michael Gummel= t <
>> >> mgummelt@mesosp= here.io>
>> >> >> wrote:
>> >> >>>
>> >> >>> >=C2=A0 I should preassume that No of exe= cutors should be less than
>> >> >>> > number
>> >> of
>> >> >>> > tasks.
>> >> >>>
>> >> >>> No.=C2=A0 Each executor runs 0 or more tasks= .
>> >> >>>
>> >> >>> Each executor consumes 1 CPU, and each task = running on that
>> >> >>> executor
>> >> >>> consumes another CPU.=C2=A0 You can customiz= e this via
>> >> >>> spark.mesos.mesosExecutor.cores
>> >> >>>
>> >> >>> (= https://github.com/apache/spark/blob/v1.6.3/docs/running-on-mesos= .md)
>> >> and
>> >> >>> spark.task.cpus
>> >> >>> (htt= ps://github.com/apache/spark/blob/v1.6.3/docs/configuration.md)
>> >> >>>
>> >> >>> On Mon, Dec 19, 2016 at 12:09 PM, Chawla,Sum= it
>> >> >>> <sumitkchawla@gmail.com
>> >> >
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Ah thanks. looks like i skipped reading = this "Neither will
>> >> >>>> executors
>> >> >>>> terminate when they=E2=80=99re idle.&quo= t;
>> >> >>>>
>> >> >>>> So in my job scenario,=C2=A0 I should pr= eassume that No of executors
>> >> >>>> should
>> >> >>>> be less than number of tasks. Ideally on= e executor should execute
>> >> >>>> 1
>> >> or more
>> >> >>>> tasks.=C2=A0 But i am observing somethin= g strange instead.=C2=A0 I start my
>> >> >>>> job
>> >> with
>> >> >>>> 48 partitions for a spark job. In mesos = ui i see that number of
>> >> >>>> tasks
>> >> is 48,
>> >> >>>> but no. of CPUs is 78 which is way more = than 48.=C2=A0 Here i am
>> >> >>>> assuming
>> >> that 1
>> >> >>>> CPU is 1 executor.=C2=A0 =C2=A0I am not = specifying any configuration to set
>> >> number of
>> >> >>>> cores per executor.
>> >> >>>>
>> >> >>>> Regards
>> >> >>>> Sumit Chawla
>> >> >>>>
>> >> >>>>
>> >> >>>> On Mon, Dec 19, 2016 at 11:35 AM, Joris = Van Remoortere
>> >> >>>> <joris@mesosphere.io> wrote:
>> >> >>>>>
>> >> >>>>> That makes sense. From the documenta= tion it looks like the
>> >> >>>>> executors
>> >> >>>>> are not supposed to terminate:
>> >> >>>>>
>> >> >>>>> htt= p://spark.apache.org/docs/latest/running-on-mesos.html#
>> >> fine-grained-deprecated
>> >> >>>>>>
>> >> >>>>>> Note that while Spark tasks in f= ine-grained will relinquish
>> >> >>>>>> cores as
>> >> >>>>>> they terminate, they will not re= linquish memory, as the JVM does
>> >> not give
>> >> >>>>>> memory back to the Operating Sys= tem. Neither will executors
>> >> terminate when
>> >> >>>>>> they=E2=80=99re idle.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> I suppose your task to executor CPU = ratio is low enough that it
>> >> >>>>> looks
>> >> >>>>> like most of the resources are not b= eing reclaimed. If your tasks
>> >> were using
>> >> >>>>> significantly more CPU the amortized= cost of the idle executors
>> >> would not be
>> >> >>>>> such a big deal.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> =E2=80=94
>> >> >>>>> Joris Van Remoortere
>> >> >>>>> Mesosphere
>> >> >>>>>
>> >> >>>>> On Mon, Dec 19, 2016 at 11:26 AM, Ti= mothy Chen
>> >> >>>>> <tnachen@gmail.com>
>> >> >>>>> wrote:
>> >> >>>>>>
>> >> >>>>>> Hi Chawla,
>> >> >>>>>>
>> >> >>>>>> One possible reason is that Meso= s fine grain mode also takes up
>> >> cores
>> >> >>>>>> to run the executor per host, so= if you have 20 agents running
>> >> >>>>>> Fine
>> >> >>>>>> grained executor it will take up= 20 cores while it's still
>> >> >>>>>> running.
>> >> >>>>>>
>> >> >>>>>> Tim
>> >> >>>>>>
>> >> >>>>>> On Fri, Dec 16, 2016 at 8:41 AM,= Chawla,Sumit <
>> >> sumitkchawla@gm= ail.com>
>> >> >>>>>> wrote:
>> >> >>>>>> > Hi
>> >> >>>>>> >
>> >> >>>>>> > I am using Spark 1.6. I hav= e one query about Fine Grained
>> >> >>>>>> > model in
>> >> >>>>>> > Spark.
>> >> >>>>>> > I have a simple Spark appli= cation which transforms A -> B.
>> >> >>>>>> > Its a
>> >> >>>>>> > single
>> >> >>>>>> > stage application.=C2=A0 To= begin the program, It starts with 48
>> >> >>>>>> > partitions.
>> >> >>>>>> > When the program starts run= ning, in mesos UI it shows 48 tasks
>> >> >>>>>> > and
>> >> >>>>>> > 48 CPUs
>> >> >>>>>> > allocated to job.=C2=A0 Now= as the tasks get done, the number of
>> >> >>>>>> > active
>> >> >>>>>> > tasks
>> >> >>>>>> > number starts decreasing.= =C2=A0 How ever, the number of CPUs does
>> >> >>>>>> > not
>> >> >>>>>> > decrease
>> >> >>>>>> > propotionally.=C2=A0 When t= he job was about to finish, there was a
>> >> single
>> >> >>>>>> > remaininig task, however CP= U count was still 20.
>> >> >>>>>> >
>> >> >>>>>> > My questions, is why there = is no one to one mapping between
>> >> >>>>>> > tasks
>> >> >>>>>> > and cpus
>> >> >>>>>> > in Fine grained?=C2=A0 How = can these CPUs be released when the job
>> >> >>>>>> > is
>> >> >>>>>> > done, so
>> >> >>>>>> > that other jobs can start.<= br> >> >> >>>>>> >
>> >> >>>>>> >
>> >> >>>>>> > Regards
>> >> >>>>>> > Sumit Chawla
>> >> >>>>>
>> >> >>>>>
>> >> >>>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Michael Gummelt
>> >> >>> Software Engineer
>> >> >>> Mesosphere
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Michael Gummelt
>> >> > Software Engineer
>> >> > Mesosphere
>> >>
>>
>>
>>
>> --
>>=C2=A0 - Davies
>
>
>
>
> --
> Michael Gummelt
> Software Engineer
> Mesosphere



--
Michael Gummelt
Software = Engineer
Mesosphere
--94eb2c1230c4787f7d05449613f2--