crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: Custom Crunch Target and Counters
Date Fri, 20 Nov 2015 18:44:31 GMT
Yeah, that does seem sort of crappy. Let me see what I can do.

On Thu, Nov 19, 2015 at 10:39 PM, <> wrote:

> Hello,
> So we wrote a custom Crunch Target to write data to a particular location
> which included providing RecordWriters, OutputFormat, OutCommitters and
> such. Now we  wanted to add Counters to get a count of how much data our
> reducers are writing. A most obvious design was to use the
> TaskAttemptContext that gets passed to the RecordWriter and use the
> getCounter() method on it to manipulate the counters. However, that did not
> work as expected and even though the counters were getting incremented,
> they did not show up on the Resource Manager UI. On further investigation I
> found that, cruch mangles this context object to add in a named output via
> the CrunchOutputs class[1] which basically makes the counters useless
> within the recordwriter class.
> Would it be a feasible enhancement for the CrunchOutputs to pass in the
> original base context object along with the modified one with named outputs
> so that it can be used for counters? Any other suggestions are most welcome
> as well.
> Thanks,
> Swarnim
> [1]

View raw message