hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Leon Mergen <l.p.mer...@solatis.com>
Subject RE: Practical limit on emitted map/reduce values
Date Thu, 18 Jun 2009 17:44:23 GMT
Hello Owen,

> > Could you perhaps elaborate on that 100 MB limit ? Is that due to a
> > limit that is caused by the Java VM heap size ? If so, could that,
> > for example, be increased to 512MB by setting mapred.child.java.opts
> > to '-Xmx512m' ?
> 
> A couple of points:
>    1. The 100MB was just for ballpark calculations. Of course if you
> have a large heap, you can fit larger values. Don't forget that the
> framework is allocating big chunks of the heap for its own buffers,
> when figuring out how big to make your heaps.
>    2. Having large keys is much harder than large values. When doing a
> N-way merge, the framework has N+1 keys and 1 value in memory at a
> time.

Ok, that makes sense. Thanks for the information!


Regards,

Leon Mergen



Mime
View raw message