crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: [jira] [Updated] (CRUNCH-209) Jobs with large numbers of directory inputs will fail with odd inputsplit exceptions
Date Thu, 23 May 2013 20:00:56 GMT

In all likelihood, we will add them back in MR2 to avoid MR AM crashes. The only  question
in when.

Thanks,
+Vinod

On May 23, 2013, at 12:57 PM, Harsh J wrote:

> There is a limit in MR1, mapred.user.jobconf.limit per
> http://hadoop.apache.org/docs/stable/mapred-default.html, that limits it to
> 5 MB (but this is applied at the JT level). I am not aware of any
> serialization-time limits and think there are none as I've seen Hive use
> the same code to write enormous sized files.
> 
> Worth noting that MR2, suitable to its service-less architecture, has no
> such limits on jobconf size and the property isn't present in it anymore.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message