crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dominique Dierickx (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-330) Use of multiple output counters can be disabled in configuration.
Date Thu, 23 Jan 2014 18:32:40 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dominique Dierickx updated CRUNCH-330:
--------------------------------------

    Description: 
We're having some trouble with the amount of counters that Crunch creates
when writing to a lot of different output files (slightly more than 120).
This wouldn't be an issue if we were able to configure the maximum number
of allowed counters but unfortunately, because we are running an older
version of Hadoop, doing this is not an option and we are required to patch
Crunch locally when using a new release to leave out the counters.

I'm not saying the counters should be removed but maybe it is an option to
make them configurable without paying too much of a performance penalty?

I will implement this functionality and submit a patch.

  was:
We're having some trouble with the amount of counters that Crunch creates
when writing to a lot of different output files (slightly more than 120).
This wouldn't be an issue if we were able to configure the maximum number
of allowed counters but unfortunately, because we are running an older
version of Hadoop, doing this is not an option and we are required to patch
Crunch locally when using a new release to leave out the counters. The
required patch (one line...) can be found in the attachment.

I'm not saying the counters should be removed but maybe it is an option to
make them configurable without paying too much of a performance penalty?

I will implement this functionality and submit a patch.


> Use of multiple output counters can be disabled in configuration.
> -----------------------------------------------------------------
>
>                 Key: CRUNCH-330
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-330
>             Project: Crunch
>          Issue Type: New Feature
>          Components: Core, IO
>            Reporter: Dominique Dierickx
>            Assignee: Josh Wills
>            Priority: Minor
>
> We're having some trouble with the amount of counters that Crunch creates
> when writing to a lot of different output files (slightly more than 120).
> This wouldn't be an issue if we were able to configure the maximum number
> of allowed counters but unfortunately, because we are running an older
> version of Hadoop, doing this is not an option and we are required to patch
> Crunch locally when using a new release to leave out the counters.
> I'm not saying the counters should be removed but maybe it is an option to
> make them configurable without paying too much of a performance penalty?
> I will implement this functionality and submit a patch.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message