hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leon Mergen <l.p.mer...@solatis.com>
Subject RE: Practical limit on emitted map/reduce values
Date Thu, 18 Jun 2009 15:45:37 GMT
Hello Owen,

> Keys and values can be large. They are certainly capped above by
> Java's 2GB limit on byte arrays. More practically, you will have
> problems running out of memory with keys or values of 100 MB. There is
> no restriction that a key/value pair fits in a single hdfs block, but
> performance would suffer. (In particular, the FileInputFormats split
> at block sized chunks, which means you will have maps that scan an
> entire block without processing anything.)

Thanks for the quick reply.

Could you perhaps elaborate on that 100 MB limit ? Is that due to a limit that is caused by
the Java VM heap size ? If so, could that, for example, be increased to 512MB by setting mapred.child.java.opts
to '-Xmx512m' ?


Leon Mergen

View raw message