crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Created] (CRUNCH-292) Hack around Hadoop2's job counter limits
Date Fri, 01 Nov 2013 16:57:27 GMT
Josh Wills created CRUNCH-292:

             Summary: Hack around Hadoop2's job counter limits
                 Key: CRUNCH-292
             Project: Crunch
          Issue Type: Bug
          Components: Core
    Affects Versions: 0.7.0
            Reporter: Josh Wills
            Assignee: Josh Wills

Hadoop2 introduces limits in the Counters library that set a maximum of 120 counters per job.
These limits are really hard to hack around (for some good reasons); the only real way to
override them is to update mapred-site.xml and restart the cluster.

This presents a challenge for Crunch's in-memory implementation, which uses the Counters library
in local mode and can potentially generate well more than 120 counters when testing long pipelines.

This message was sent by Atlassian JIRA

View raw message