crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: [jira] [Updated] (CRUNCH-209) Jobs with large numbers of directory inputs will fail with odd inputsplit exceptions
Date Thu, 23 May 2013 20:04:27 GMT
Thanks guys, good to know.


On Thu, May 23, 2013 at 1:00 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

>
> In all likelihood, we will add them back in MR2 to avoid MR AM crashes.
> The only  question in when.
>
> Thanks,
> +Vinod
>
> On May 23, 2013, at 12:57 PM, Harsh J wrote:
>
> > There is a limit in MR1, mapred.user.jobconf.limit per
> > http://hadoop.apache.org/docs/stable/mapred-default.html, that limits
> it to
> > 5 MB (but this is applied at the JT level). I am not aware of any
> > serialization-time limits and think there are none as I've seen Hive use
> > the same code to write enormous sized files.
> >
> > Worth noting that MR2, suitable to its service-less architecture, has no
> > such limits on jobconf size and the property isn't present in it anymore.
>
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message