mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: mapreduce memory issues
Date Wed, 05 May 2010 21:26:17 GMT
On Wed, May 5, 2010 at 8:54 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
> One big win would be to use Benson's collections instead of a normal map
> (you may already have this).

Of course, already doing so !


> Another problem is the expectation that a map-reduce should be happy with
> 500MB of memory.  I don't think that is a universally reasonable

Agree with this sentiment; I might pick a somewhat higher value as the
reasonable max memory for a worker... but I came from Google land
where RAM grows on trees. See below for more on this.


> common than others.  Regardless, however, it is reasonable to expect that if
> you see about the same amount of data that you will see about the same
> number of items.  In a reasonably scaling system, the amount of data each
> node in the cluster sees will be roughly constant as the input and cluster
> size scale together.  In a map-reduce architecture, you even get this
> property if the input size and number of mappers and reducers scale
> together.

... which is the assumption that made me guess this lookup table isn't
going to get big. But if I'm right, it doesn't hurt to cap the size of
the lookup. Which I can commit. But maybe that's not the issue either.


> One observation that makes me suspect that the problem is not the size of
> this table that Sean has mentioned is the fact that Tamas has observed that
> he doesn't get more tasks when he scales the cluster size.  That may be that
> Hadoop is happy with the number of splits and has to be forced to split more
> (by setting max split size or number of mappers).   The problem might also
> be that this program as designed just needs 3GB to complete and efforts to
> force it into 500MB will always fail.

Tamas, are you configuring the number of mappers and reducers with
command line args? It will not by default run many workers.

I don't know of any stage of this computation that should require
significant memory, so it just can't need this much memory. Besides
this little lookup map, the worst thing it does is load a whole row of
the co-occurrence matrix in memory.


Actually I think I know the answer, perhaps. This is one of two stages
that relies heavily on the combiner for efficiency. It sets io.sort.mb
(the size of the buffer used to hold map output records before
combining and spilling to disk) high. I suspect the issue is here
since it seems to fail when allocating space for the map output
somehow.

Its value is 1000MB, or, if you've set a heap size via
mapred.child.java.opts, half of the heap size you allocate. It
actually does this via a crude regex looking for "Xmx[0-9]+m" in the
string.

So I can imagine that if you don't set it, this buffer overruns the
heap. Or if you do set it, low, then other memory usage overruns half
the heap and it fails again at some point.

You are welcome to test this by modifying setIOSort() in
RecommenderJob. Just crank down that "1000", or divide heapMB by a
larger value. This should let you run with somewhat smaller heap
sizes. If that's the culprit I'm happy to turn down these values a
bit.

Mime
View raw message