crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <mkwhita...@gmail.com>
Subject Re: Custom Crunch Target and Counters
Date Mon, 23 Nov 2015 00:09:04 GMT
Logged this request here to track progress...
https://issues.apache.org/jira/browse/CRUNCH-579

On Fri, Nov 20, 2015 at 12:44 PM, Josh Wills <josh.wills@gmail.com> wrote:

> Yeah, that does seem sort of crappy. Let me see what I can do.
>
> On Thu, Nov 19, 2015 at 10:39 PM, kulkarni.swarnim@gmail.com <
> kulkarni.swarnim@gmail.com> wrote:
>
>> Hello,
>>
>> So we wrote a custom Crunch Target to write data to a particular location
>> which included providing RecordWriters, OutputFormat, OutCommitters and
>> such. Now we  wanted to add Counters to get a count of how much data our
>> reducers are writing. A most obvious design was to use the
>> TaskAttemptContext that gets passed to the RecordWriter and use the
>> getCounter() method on it to manipulate the counters. However, that did not
>> work as expected and even though the counters were getting incremented,
>> they did not show up on the Resource Manager UI. On further investigation I
>> found that, cruch mangles this context object to add in a named output via
>> the CrunchOutputs class[1] which basically makes the counters useless
>> within the recordwriter class.
>>
>> Would it be a feasible enhancement for the CrunchOutputs to pass in the
>> original base context object along with the modified one with named outputs
>> so that it can be used for counters? Any other suggestions are most welcome
>> as well.
>>
>> Thanks,
>> Swarnim
>>
>> [1]
>> https://github.com/cloudera/crunch/blob/cdh5.4.8-release/crunch-core/src/main/java/org/apache/crunch/io/CrunchOutputs.java#L210-L232
>>
>
>

Mime
View raw message