crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-292) Hack around Hadoop2's job counter limits
Date Fri, 01 Nov 2013 17:07:21 GMT


Josh Wills updated CRUNCH-292:

    Attachment: CRUNCH-292.patch

Here's a patch to fix it by dynamically creating new instances of Counters whenever we hit
an exception creating a counter. The only hitch is that I can't really return the CounterGroup
for a given group name using this approach, but I hope that doesn't cause too much of an issue
for clients.

> Hack around Hadoop2's job counter limits
> ----------------------------------------
>                 Key: CRUNCH-292
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-292.patch
> Hadoop2 introduces limits in the Counters library that set a maximum of 120 counters
per job. These limits are really hard to hack around (for some good reasons); the only real
way to override them is to update mapred-site.xml and restart the cluster.
> This presents a challenge for Crunch's in-memory implementation, which uses the Counters
library in local mode and can potentially generate well more than 120 counters when testing
long pipelines. 

This message was sent by Atlassian JIRA

View raw message