crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-292) Hack around Hadoop2's job counter limits
Date Fri, 01 Nov 2013 20:53:17 GMT


Josh Wills updated CRUNCH-292:

    Attachment: CRUNCH-292b.patch

Small tweak to the getGroup(String name) impl in the wrapper to make it work in most normal
circumstances. Planning on committing this one later today.

> Hack around Hadoop2's job counter limits
> ----------------------------------------
>                 Key: CRUNCH-292
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-292.patch, CRUNCH-292b.patch
> Hadoop2 introduces limits in the Counters library that set a maximum of 120 counters
per job. These limits are really hard to hack around (for some good reasons); the only real
way to override them is to update mapred-site.xml and restart the cluster.
> This presents a challenge for Crunch's in-memory implementation, which uses the Counters
library in local mode and can potentially generate well more than 120 counters when testing
long pipelines. 

This message was sent by Atlassian JIRA

View raw message