crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <>
Subject Re: [jira] [Updated] (CRUNCH-209) Jobs with large numbers of directory inputs will fail with odd inputsplit exceptions
Date Thu, 23 May 2013 20:00:56 GMT

In all likelihood, we will add them back in MR2 to avoid MR AM crashes. The only  question
in when.


On May 23, 2013, at 12:57 PM, Harsh J wrote:

> There is a limit in MR1, mapred.user.jobconf.limit per
>, that limits it to
> 5 MB (but this is applied at the JT level). I am not aware of any
> serialization-time limits and think there are none as I've seen Hive use
> the same code to write enormous sized files.
> Worth noting that MR2, suitable to its service-less architecture, has no
> such limits on jobconf size and the property isn't present in it anymore.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message