crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-292) Hack around Hadoop2's job counter limits
Date Fri, 01 Nov 2013 17:05:17 GMT


Vinod Kumar Vavilapalli commented on CRUNCH-292:

These limits are per job, and MR is a user land library now anyways, so you don't need to
restart YARN?

> Hack around Hadoop2's job counter limits
> ----------------------------------------
>                 Key: CRUNCH-292
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7.0
>            Reporter: Josh Wills
>            Assignee: Josh Wills
> Hadoop2 introduces limits in the Counters library that set a maximum of 120 counters
per job. These limits are really hard to hack around (for some good reasons); the only real
way to override them is to update mapred-site.xml and restart the cluster.
> This presents a challenge for Crunch's in-memory implementation, which uses the Counters
library in local mode and can potentially generate well more than 120 counters when testing
long pipelines. 

This message was sent by Atlassian JIRA

View raw message