crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nithin Asokan <>
Subject Spark accumulators
Date Thu, 03 Sep 2015 12:25:25 GMT
We are currently testing a few capabilities using Spark and one thing we
noticed in Spark is they don't list any user defined accumulators on web

On MapReduce I would imagine counters being displayed on the job page,
however on a SparkPipeline I was only able to pull counter information from

I think the reason these accumulators are not visible on web UI is because
crunch does not name these accumulators. Spark expects an accumulator to
have a name to be visible on the UI.
API with Name)

I would like to know if it's possible in crunch to name these accumulators
so they are available in web UI. This will give us an experience where
users can monitor/watch accumulators from web UI to obtain key information
about their jobs.


View raw message