crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <>
Subject Re: Spark accumulators
Date Thu, 03 Sep 2015 12:49:43 GMT
Nithin logged to track
this work.  Also added a patch for the change you proposed, want to try it
out and see if that works for you?

On Thu, Sep 3, 2015 at 7:25 AM, Nithin Asokan <> wrote:

> We are currently testing a few capabilities using Spark and one thing we
> noticed in Spark is they don't list any user defined accumulators on web
> UI.
> On MapReduce I would imagine counters being displayed on the job page,
> however on a SparkPipeline I was only able to pull counter information from
> PipelineResult#getStageResult().
> I think the reason these accumulators are not visible on web UI is because
> crunch does not name these accumulators. Spark expects an accumulator to
> have a name to be visible on the UI.
> API with Name)
> I would like to know if it's possible in crunch to name these accumulators
> so they are available in web UI. This will give us an experience where
> users can monitor/watch accumulators from web UI to obtain key information
> about their jobs.
> Thanks,
> Nithin

View raw message