crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: Too many Crunch output counters
Date Thu, 16 Jan 2014 18:27:32 GMT
That seems relatively benign; do you need a crunch param that would control
the usage of the named outputs counter?

On Thu, Jan 16, 2014 at 8:40 AM, Dierickx Dominique <>wrote:

> We're having some trouble with the amount of counters that Crunch creates
> when writing to a lot of different output files (slightly more than 120).
> This wouldn't be an issue if we were able to configure the maximum number
> of allowed counters but unfortunately, because we are running an older
> version of Hadoop, doing this is not an option and we are required to patch
> Crunch locally when using a new release to leave out the counters. The
> required patch (one line...) can be found in the attachment.
> I'm not saying the counters should be removed but maybe it is an option to
> make them configurable without paying too much of a performance penalty?
> Regards,
> Dominique Dierickx

Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message