Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 118B118ED1 for ; Tue, 27 Oct 2015 07:23:47 +0000 (UTC) Received: (qmail 92693 invoked by uid 500); 27 Oct 2015 07:23:43 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 92601 invoked by uid 500); 27 Oct 2015 07:23:43 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 92591 invoked by uid 99); 27 Oct 2015 07:23:43 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Oct 2015 07:23:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 98639C09DF for ; Tue, 27 Oct 2015 07:23:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id t18DyFUenXzL for ; Tue, 27 Oct 2015 07:23:33 +0000 (UTC) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 377C320604 for ; Tue, 27 Oct 2015 07:23:33 +0000 (UTC) Received: by lbbes7 with SMTP id es7so56737669lbb.2 for ; Tue, 27 Oct 2015 00:23:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=z8+rFMRKgGrZfWXrhanwu2SaSIG9ZyxfQbKATdD1wZk=; b=Ij8Gv0215JgBhrTWLcL9fxGh8z1BumX1K/uoSb6TVqPBB+FZ8S7OQWgqFEjj+ACXy3 3yZfJfxEVlPzzVEY9oKLgGNvkqeA48u1GS/ukUImMxGuQ1Ehd24sVaELOv/TkXqLLQvl uoyb1I9lLXplsv4v0CL0kX2CnY6099dunE3njOumsnLCIgOKDzQvhWKfRCAPIPonytLi ddcY7WwZWLdw+qFU7Vb4gVNrXkEEPXcNWQFlvGVUsFVuuZPWksX5AezpcozreHUsH4vK qkJZog2dQNDbsRO5DW/z2Wfr8xafyUAFyLYAxZXH696APr6Q6IXunXqc8j0CaCJPGyIK OwTA== MIME-Version: 1.0 X-Received: by 10.112.168.194 with SMTP id zy2mr20072404lbb.79.1445930612295; Tue, 27 Oct 2015 00:23:32 -0700 (PDT) Received: by 10.25.207.15 with HTTP; Tue, 27 Oct 2015 00:23:32 -0700 (PDT) In-Reply-To: References: Date: Tue, 27 Oct 2015 07:23:32 +0000 Message-ID: Subject: Re: Dynamic Resource Allocation with Spark Streaming (Standalone Cluster, Spark 1.5.1) From: Deenar Toraskar To: Ted Yu , user Content-Type: multipart/alternative; boundary=001a11c333f284129b052310f475 --001a11c333f284129b052310f475 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Till Spark Streaming supports dynamic allocation, you could use StreamingListener to monitor batch execution times and based on it sparkContext.requestExecutors() and sparkContext.killExecutors() to add and remove executors explicitly and . On 26 October 2015 at 21:37, Ted Yu wrote: > This is related: > SPARK-10955 Warn if dynamic allocation is enabled for Streaming jobs > > which went into 1.6.0 as well. > > FYI > > On Mon, Oct 26, 2015 at 2:26 PM, Silvio Fiorito < > silvio.fiorito@granturing.com> wrote: > >> Hi Matthias, >> >> Unless there was a change in 1.5, I'm afraid dynamic resource allocation >> is not yet supported in streaming apps. >> >> Thanks, >> Silvio >> >> Sent from my Lumia 930 >> ------------------------------ >> From: Matthias Niehoff >> Sent: =E2=80=8E10/=E2=80=8E26/=E2=80=8E2015 4:00 PM >> To: user@spark.apache.org >> Subject: Dynamic Resource Allocation with Spark Streaming (Standalone >> Cluster, Spark 1.5.1) >> >> Hello everybody, >> >> I have a few (~15) Spark Streaming jobs which have load peaks as well as >> long times with a low load. So I thought the new Dynamic Resource >> Allocation for Standalone Clusters might be helpful (SPARK-4751). >> >> I have a test "cluster" with 1 worker consisting of 4 executors with 2 >> cores each, so 8 cores in total. >> >> I started a simple streaming application without limiting the max cores >> for this app. As expected the app occupied every core of the cluster. Th= en >> I started a second app, also without limiting the maximum cores. As the >> first app did not get any input through the stream, my naive expectation >> was that the second app would get at least 2 cores (1 receiver, 1 >> processing), but that's not what happened. The cores are still assigned = to >> the first app. >> When I look at the application UI of the first app every executor is >> still running. That explains why no executor is used for the second app. >> >> I end up with two questions: >> - When does an executor getting idle in a Spark Streaming application? >> (and so could be reassigned to another app) >> - Is there another way to compete with uncertain load when using Spark >> Streaming Applications? I already combined multiple jobs to a Spark >> Application using different threads, but this approach comes to a limit = for >> me, because Spark Applications get to big to manage. >> >> Thank You! >> >> >> > --001a11c333f284129b052310f475 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Till Spark Streaming supports dynamic allocation, you could use=C2= =A0StreamingListener=C2=A0to monitor batch execution times and based o= n it=C2=A0sparkContext.requestExecu= tors() and sparkContext.killExecuto= rs() to add and remove executors explicitly and=C2=A0.


=

On 26= October 2015 at 21:37, Ted Yu <yuzhihong@gmail.com> wrote= :
This is related:
S= PARK-10955=C2=A0Warn if dynamic allocation is enabled for Streaming jobs

which went into 1.6.0 as well.

FYI

On Mon, Oct 26, 2015 at 2:2= 6 PM, Silvio Fiorito <silvio.fiorito@granturing.com> wrote:
Hi Matthias,
Unless there was a change in 1.5, I'm afraid dynamic resource allocatio= n is not yet supported in streaming apps.

Thanks,
Silvio

Sent from my Lumia 930
Hello everybody,

I have a few (~15) Spark Streaming jobs which have load peaks as well = as long times with a low load. So I thought the new Dynamic Resource Alloca= tion for Standalone Clusters might be helpful (SPARK-4751).

I have a test "cluster" with 1 worker consisting of 4 execut= ors with 2 cores each, so 8 cores in total.

I started a simple streaming application without limiting the max core= s for this app. As expected the app occupied every core of the cluster. The= n I started a second app, also without limiting the maximum cores. As the f= irst app did not get any input through the stream, my naive expectation was that the second app would get at leas= t 2 cores (1 receiver, 1 processing), but that's not what happened. The= cores are still assigned to the first app.
When I look at the application UI of the first app every executor is s= till running. That explains why no executor is used for the second app.=C2= =A0

I end up with two questions:
- When does an executor getting idle in a Spark Streaming application?= (and so could be reassigned to another app)
- Is there another way to compete with uncertain load when using Spark= Streaming Applications? I already combined multiple jobs to a Spark Applic= ation using different threads, but this approach comes to a limit for me, b= ecause Spark Applications get to big to manage.

Thank You!




--001a11c333f284129b052310f475--