crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-292) Hack around Hadoop2's job counter limits
Date Fri, 01 Nov 2013 17:09:23 GMT


Josh Wills commented on CRUNCH-292:

@Vinod I just meant that the limits are specified in the Counters library itself (,
specifically), which the in-memory pipeline uses directly. No YARN/MR issues here, it's easy
to get around it on the cluster itself as you indicated.

> Hack around Hadoop2's job counter limits
> ----------------------------------------
>                 Key: CRUNCH-292
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-292.patch
> Hadoop2 introduces limits in the Counters library that set a maximum of 120 counters
per job. These limits are really hard to hack around (for some good reasons); the only real
way to override them is to update mapred-site.xml and restart the cluster.
> This presents a challenge for Crunch's in-memory implementation, which uses the Counters
library in local mode and can potentially generate well more than 120 counters when testing
long pipelines. 

This message was sent by Atlassian JIRA

View raw message