mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Eastman <jeast...@Narus.com>
Subject RE: ClusterDumper issue
Date Mon, 31 Oct 2011 16:17:20 GMT
Unfortunately, the cluster dumper loads all the points into memory so it can sort them by cluster
for display. What are you trying to do with the 20M points? Certainly not display them! A
better step for subsequent processing would be for you to write a short MR program to read
in the clusteredPoints directory and output each point to its clustered (mapper). Then, using
k reducers, each of which will get the points for one cluster, each reducer will output a
folder containing all the points for that cluster.

-----Original Message-----
From: Vitaly Davydov [mailto:vdavydov@mirantis.com] 
Sent: Monday, October 31, 2011 4:21 AM
To: user@mahout.apache.org
Subject: ClusterDumper issue

Hello,
We use mahout Kmeans-algorithm and convert its binary output to text
representation via ClusterDumper. When our input has reached approximately
20 million points, ClusterDumper takes 2.5 Gb RAM and fails with "Out of
memory error". Our machines don't have swap and we can't increase RAM
currently. Is there a way to avoid this problem?

The exception is attached below:

"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at
org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:101)
at
org.apache.mahout.common.iterator.sequencefile.SequenceFileIterator.computeNext(SequenceFileIterator.java:38)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:474)
at
com.google.common.collect.ForwardingIterator.hasNext(ForwardingIterator.java:39)
at
org.apache.mahout.utils.clustering.ClusterDumper.readPoints(ClusterDumper.java:239)
at
org.apache.mahout.utils.clustering.ClusterDumper.init(ClusterDumper.java:193)
at
org.apache.mahout.utils.clustering.ClusterDumper.<init>(ClusterDumper.java:78)
at com.mirantis.bigdata.clustering.kmeans.KmeansJob.run(Unknown Source)
at com.mirantis.bigdata.clustering.kmeans.KmeansJob.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)"

-- 
Regards,
Vitaly Davydov

Mime
View raw message