crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: Too many Crunch output counters
Date Thu, 16 Jan 2014 18:27:32 GMT
That seems relatively benign; do you need a crunch param that would control
the usage of the named outputs counter?


On Thu, Jan 16, 2014 at 8:40 AM, Dierickx Dominique <d.dierickx@gmail.com>wrote:

> We're having some trouble with the amount of counters that Crunch creates
> when writing to a lot of different output files (slightly more than 120).
> This wouldn't be an issue if we were able to configure the maximum number
> of allowed counters but unfortunately, because we are running an older
> version of Hadoop, doing this is not an option and we are required to patch
> Crunch locally when using a new release to leave out the counters. The
> required patch (one line...) can be found in the attachment.
>
> I'm not saying the counters should be removed but maybe it is an option to
> make them configurable without paying too much of a performance penalty?
>
> Regards,
> Dominique Dierickx
>
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message