Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
References: 
 <CABafDq29PaUcYyUw47f7F782RrTRjqpNYvUmDK9bkb5LMG4GaA@mail.gmail.com>
 <CADH-_NJL1OzxLjQnR4i_Eojc6ymzi1nM5Nxf9gk4D396taYtCw@mail.gmail.com>
In-Reply-To: 
 <CADH-_NJL1OzxLjQnR4i_Eojc6ymzi1nM5Nxf9gk4D396taYtCw@mail.gmail.com>
From: Nithin Asokan <anithin19@gmail.com>
Date: Thu, 03 Sep 2015 18:08:11 +0000
Message-ID: 
 <CABafDq3me=DdgaDCFGQaEcsRkG4npTKG2oCPkdY1tRHnHVVUmg@mail.gmail.com>
Subject: Re: Spark accumulators
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0357501caf9b051edbabfa

--94eb2c0357501caf9b051edbabfa
Content-Type: text/plain; charset=UTF-8

Thank you Micah. With the patch I'm able to see accumulators listed on my
Spark application.

On Thu, Sep 3, 2015 at 7:49 AM Micah Whitacre <mkwhitacre@gmail.com> wrote:

> Nithin logged https://issues.apache.org/jira/browse/CRUNCH-558 to track
> this work.  Also added a patch for the change you proposed, want to try it
> out and see if that works for you?
>
> On Thu, Sep 3, 2015 at 7:25 AM, Nithin Asokan <anithin19@gmail.com> wrote:
>
>> We are currently testing a few capabilities using Spark and one thing we
>> noticed in Spark is they don't list any user defined accumulators on web
>> UI.
>>
>> On MapReduce I would imagine counters being displayed on the job page,
>> however on a SparkPipeline I was only able to pull counter information from
>> PipelineResult#getStageResult().
>>
>> I think the reason these accumulators are not visible on web UI is
>> because crunch does not name these accumulators. Spark expects an
>> accumulator to have a name to be visible on the UI.
>>
>>
>> https://github.com/apache/crunch/blob/apache-crunch-0.13.0/crunch-spark/src/main/java/org/apache/crunch/impl/spark/SparkRuntime.java#L125-L126
>>
>>
>> https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala#L616-L624 (accumulator
>> API with Name)
>>
>> I would like to know if it's possible in crunch to name these
>> accumulators so they are available in web UI. This will give us an
>> experience where users can monitor/watch accumulators from web UI to obtain
>> key information about their jobs.
>>
>> Thanks,
>> Nithin
>>
>
>

--94eb2c0357501caf9b051edbabfa
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thank you Micah. With the patch I&#39;m able to see accumu=
lators listed on my Spark application. =C2=A0<br><br><div class=3D"gmail_qu=
ote"><div dir=3D"ltr">On Thu, Sep 3, 2015 at 7:49 AM Micah Whitacre &lt;<a =
href=3D"mailto:mkwhitacre@gmail.com">mkwhitacre@gmail.com</a>&gt; wrote:<br=
></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-=
left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">Nithin logged=C2=A0<=
a href=3D"https://issues.apache.org/jira/browse/CRUNCH-558" target=3D"_blan=
k">https://issues.apache.org/jira/browse/CRUNCH-558</a> to track this work.=
=C2=A0 Also added a patch for the change you proposed, want to try it out a=
nd see if that works for you?</div><div class=3D"gmail_extra"><br><div clas=
s=3D"gmail_quote">On Thu, Sep 3, 2015 at 7:25 AM, Nithin Asokan <span dir=
=3D"ltr">&lt;<a href=3D"mailto:anithin19@gmail.com" target=3D"_blank">anith=
in19@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div=
 dir=3D"ltr"><div>We are currently testing a few capabilities using Spark a=
nd one thing we noticed in Spark is they don&#39;t list any user defined ac=
cumulators on web UI.=C2=A0<br></div><div><br></div><div>On MapReduce I wou=
ld imagine counters being displayed on the job page, however on a SparkPipe=
line I was only able to pull counter information from PipelineResult#getSta=
geResult().=C2=A0<br></div><div><br></div><div>I think the reason these acc=
umulators are not visible on web UI is because crunch does not name these a=
ccumulators. Spark expects an accumulator to have a name to be visible on t=
he UI.</div><div><br></div><div><a href=3D"https://github.com/apache/crunch=
/blob/apache-crunch-0.13.0/crunch-spark/src/main/java/org/apache/crunch/imp=
l/spark/SparkRuntime.java#L125-L126" target=3D"_blank">https://github.com/a=
pache/crunch/blob/apache-crunch-0.13.0/crunch-spark/src/main/java/org/apach=
e/crunch/impl/spark/SparkRuntime.java#L125-L126</a><br></div><div><br></div=
><div><a href=3D"https://github.com/apache/spark/blob/v1.4.1/core/src/main/=
scala/org/apache/spark/api/java/JavaSparkContext.scala#L616-L624" target=3D=
"_blank">https://github.com/apache/spark/blob/v1.4.1/core/src/main/scala/or=
g/apache/spark/api/java/JavaSparkContext.scala#L616-L624</a>=C2=A0(accumula=
tor API with Name)<br></div><div><br></div><div>I would like to know if it&=
#39;s possible in crunch to name these accumulators so they are available i=
n web UI. This will give us an experience where users can monitor/watch acc=
umulators from web UI to obtain key information about their jobs.=C2=A0</di=
v><div><br></div><div>Thanks,</div><div>Nithin</div></div>
</blockquote></div><br></div>
</blockquote></div></div>

--94eb2c0357501caf9b051edbabfa--