Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (nike.apache.org: domain of manojsameltech@gmail.com
 designates 209.85.212.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAOnQ7uK8MEqY0jZv511yvY5YgF=t7GvC=hF_xV5XVZ2y--POQ@mail.gmail.com>
References: 
 <CAAqMD1j7_A62YZvWbxpkzax77x1KLWcJOa8XexeHvVEKoJ+4rw@mail.gmail.com>
	<CAAqMD1jyXmU03FrV_xYMJgoY-aHy0=mgQbLNZYwY_55Tnc-RTg@mail.gmail.com>
	<CALte62xTFWT3D7nGFTOjiESA6xa_WmzPxK=bF8V8XygFVw+TZA@mail.gmail.com>
	<CAAqMD1ggfpsot9jzNMwyghNetKRF_h_Ycz_cWCtTHUqZ=7fjAA@mail.gmail.com>
	<CAAOnQ7uK8MEqY0jZv511yvY5YgF=t7GvC=hF_xV5XVZ2y--POQ@mail.gmail.com>
Date: Mon, 23 Mar 2015 14:51:51 -0700
Message-ID: 
 <CAAqMD1ifQovmerMvi243o4A9vLNogeRhRrOBc=_MAyLPxZFhhw@mail.gmail.com>
Subject: Re: Spark 1.3 Dynamic Allocation - Requesting 0 new executor(s)
 because tasks are backlogged
From: Manoj Samel <manojsameltech@gmail.com>
To: Marcelo Vanzin <vanzin@cloudera.com>
Cc: "user@spark.apache.org" <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=f46d0435c0347c5a460511fbacc2

--f46d0435c0347c5a460511fbacc2
Content-Type: text/plain; charset=UTF-8

Log shows stack traces that seem to match the assert in JIRA so it seems I
am hitting the issue. Thanks for the heads up ...

15/03/23 20:29:50 ERROR actor.OneForOneStrategy: assertion failed:
Allocator killed more executors than are allocated!
java.lang.AssertionError: assertion failed: Allocator killed more executors
than are allocated!
        at scala.Predef$.assert(Predef.scala:179)
        at
org.apache.spark.deploy.yarn.YarnAllocator.killExecutor(YarnAllocator.scala:152)
        at
org.apache.spark.deploy.yarn.ApplicationMaster$AMActor$$anonfun$receive$1$$anonfun$applyOrElse$6.apply(ApplicationMaster.scala:547)
        at
org.apache.spark.deploy.yarn.ApplicationMaster$AMActor$$anonfun$receive$1$$anonfun$applyOrElse$6.apply(ApplicationMaster.scala:547)
        at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at
org.apache.spark.deploy.yarn.ApplicationMaster$AMActor$$anonfun$receive$1.applyOrElse(ApplicationMaster.scala:547)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
        at
org.apache.spark.deploy.yarn.ApplicationMaster$AMActor.aroundReceive(ApplicationMaster.scala:506)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
        at akka.actor.ActorCell.invoke(ActorCell.scala:487)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
        at akka.dispatch.Mailbox.run(Mailbox.scala:220)
        at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
        at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

On Mon, Mar 23, 2015 at 2:25 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Mon, Mar 23, 2015 at 2:15 PM, Manoj Samel <manojsameltech@gmail.com>
> wrote:
> > Found the issue above error - the setting for spark_shuffle was
> incomplete.
> >
> > Now it is able to ask and get additional executors. The issue is once
> they
> > are released, it is not able to proceed with next query.
>
> That looks like SPARK-6325, which unfortunately was not fixed in time
> for 1.3.0...
>
> --
> Marcelo
>

--f46d0435c0347c5a460511fbacc2
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Log shows stack traces that seem to match the assert in JI=
RA so it seems I am hitting the issue. Thanks for the heads up ...<div><br>=
</div><div><div>15/03/23 20:29:50 ERROR actor.OneForOneStrategy: assertion =
failed: Allocator killed more executors than are allocated!</div><div>java.=
lang.AssertionError: assertion failed: Allocator killed more executors than=
 are allocated!</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scala.Predef$.asse=
rt(Predef.scala:179)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.sp=
ark.deploy.yarn.YarnAllocator.killExecutor(YarnAllocator.scala:152)</div><d=
iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.spark.deploy.yarn.ApplicationM=
aster$AMActor$$anonfun$receive$1$$anonfun$applyOrElse$6.apply(ApplicationMa=
ster.scala:547)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.spark.d=
eploy.yarn.ApplicationMaster$AMActor$$anonfun$receive$1$$anonfun$applyOrEls=
e$6.apply(ApplicationMaster.scala:547)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray=
.scala:59)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scala.collection.mutabl=
e.ArrayBuffer.foreach(ArrayBuffer.scala:47)</div><div>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 at org.apache.spark.deploy.yarn.ApplicationMaster$AMActor$$anonfun$r=
eceive$1.applyOrElse(ApplicationMaster.scala:547)</div><div>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 at akka.actor.Actor$class.aroundReceive(Actor.scala:465)</div=
><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.spark.deploy.yarn.Applicati=
onMaster$AMActor.aroundReceive(ApplicationMaster.scala:506)</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0 at akka.actor.ActorCell.receiveMessage(ActorCell.s=
cala:516)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at akka.actor.ActorCell.inv=
oke(ActorCell.scala:487)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at akka.disp=
atch.Mailbox.processMailbox(Mailbox.scala:238)</div><div>=C2=A0 =C2=A0 =C2=
=A0 =C2=A0 at akka.dispatch.Mailbox.run(Mailbox.scala:220)</div><div>=C2=A0=
 =C2=A0 =C2=A0 =C2=A0 at akka.dispatch.ForkJoinExecutorConfigurator$AkkaFor=
kJoinTask.exec(AbstractDispatcher.scala:393)</div><div>=C2=A0 =C2=A0 =C2=A0=
 =C2=A0 at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:=
260)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scala.concurrent.forkjoin.For=
kJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)</div><div>=C2=A0 =C2=A0=
 =C2=A0 =C2=A0 at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoin=
Pool.java:1979)</div></div></div><div class=3D"gmail_extra"><br><div class=
=3D"gmail_quote">On Mon, Mar 23, 2015 at 2:25 PM, Marcelo Vanzin <span dir=
=3D"ltr">&lt;<a href=3D"mailto:vanzin@cloudera.com" target=3D"_blank">vanzi=
n@cloudera.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><spa=
n class=3D"">On Mon, Mar 23, 2015 at 2:15 PM, Manoj Samel &lt;<a href=3D"ma=
ilto:manojsameltech@gmail.com">manojsameltech@gmail.com</a>&gt; wrote:<br>
&gt; Found the issue above error - the setting for spark_shuffle was incomp=
lete.<br>
&gt;<br>
&gt; Now it is able to ask and get additional executors. The issue is once =
they<br>
&gt; are released, it is not able to proceed with next query.<br>
<br>
</span>That looks like SPARK-6325, which unfortunately was not fixed in tim=
e<br>
for 1.3.0...<br>
<span class=3D"HOEnZb"><font color=3D"#888888"><br>
--<br>
Marcelo<br>
</font></span></blockquote></div><br></div>

--f46d0435c0347c5a460511fbacc2--