mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jure Jeseni─Źnik <Jure.Jesenic...@planet9.si>
Subject RE: canopy map 100% reduce 100%, then memory crazy increase
Date Tue, 23 Nov 2010 12:10:04 GMT
I probably had the same problem. 
See this thread: 
http://mail-archives.apache.org/mod_mbox/mahout-user/201011.mbox/%3C0EDE11E319B0B043B4F24E0305CABF7C80413134A4@P9MAIL.p9.internal%3E

A short answer would be that this was a bug in the current release and Jeff fixed it this
weekend. I tested yesterday and I can confirm that it works now.
Just checkout the latest version and use that instead of the bundled release. It should work.

Here are the instructions
https://cwiki.apache.org/MAHOUT/buildingmahout.html

best regards.

Jure



-----Original Message-----
From: beneo_7 [mailto:beneo_7@163.com] 
Sent: Tuesday, November 23, 2010 9:07 AM
To: user@mahout.apache.org
Subject: canopy map 100% reduce 100%, then memory crazy increase

i use the mahout 4.0 release.

in mahout-distribution-0.4/bin, i used
./mahout canopy -i /home/space/lucene_clustering/vector/vector -o /home/space/lucene_clustering/canopy/
-dm org.apache.mahout.common.distance.EuclideanDistanceMeasure -t1 0.8 -t2 0.2 -ow

int hadoop-env.sh, i add the
export HADOOP_HEAPSIZE=20000
export HADOOP_OPTS="-Xmn3g -Xss128k -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
-XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"

i am sure all the parameters are workable, because i use the jconsole to check  the vm paramters.

however, after map 100% and reduce 100%, the memory increase from 2.5G to 20G and the exception
thrown. the file vector is 30m, 50000 records, which is used for canopy.

10/11/23 16:04:27 INFO mapred.LocalJobRunner: reduce > reduce
10/11/23 16:04:27 INFO mapred.JobClient:  map 100% reduce 100%
10/11/23 16:04:30 INFO mapred.LocalJobRunner: reduce > reduce
10/11/23 16:08:17 WARN mapred.LocalJobRunner: job_local_0001
java.lang.OutOfMemoryError: Java heap space
    at org.apache.mahout.math.map.OpenIntDoubleHashMap.rehash(OpenIntDoubleHashMap.java:434)
    at org.apache.mahout.math.map.OpenIntDoubleHashMap.put(OpenIntDoubleHashMap.java:387)
    at org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:134)
    at org.apache.mahout.math.AbstractVector.assign(AbstractVector.java:449)
    at org.apache.mahout.clustering.AbstractCluster.computeParameters(AbstractCluster.java:184)
    at org.apache.mahout.clustering.canopy.CanopyReducer.reduce(CanopyReducer.java:42)
    at org.apache.mahout.clustering.canopy.CanopyReducer.reduce(CanopyReducer.java:29)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:566)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:216)
10/11/23 16:08:18 INFO mapred.JobClient: Job complete: job_local_0001
10/11/23 16:08:18 INFO mapred.JobClient: Counters: 12
10/11/23 16:08:18 INFO mapred.JobClient:   FileSystemCounters
10/11/23 16:08:18 INFO mapred.JobClient:     FILE_BYTES_READ=70413991
10/11/23 16:08:18 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=164338288
10/11/23 16:08:18 INFO mapred.JobClient:   Map-Reduce Framework
10/11/23 16:08:18 INFO mapred.JobClient:     Reduce input groups=1
10/11/23 16:08:18 INFO mapred.JobClient:     Combine output records=0
10/11/23 16:08:18 INFO mapred.JobClient:     Map input records=50000
10/11/23 16:08:18 INFO mapred.JobClient:     Reduce shuffle bytes=0
10/11/23 16:08:18 INFO mapred.JobClient:     Reduce output records=227
10/11/23 16:08:18 INFO mapred.JobClient:     Spilled Records=64708
10/11/23 16:08:18 INFO mapred.JobClient:     Map output bytes=8836211
10/11/23 16:08:18 INFO mapred.JobClient:     Combine input records=0
10/11/23 16:08:18 INFO mapred.JobClient:     Map output records=32354
10/11/23 16:08:18 INFO mapred.JobClient:     Reduce input records=32354
Exception in thread "main" java.lang.InterruptedException: Canopy Job failed processing /home/space/lucene_clustering/vector/vector
    at org.apache.mahout.clustering.canopy.CanopyDriver.buildClustersMR(CanopyDriver.java:252)
    at org.apache.mahout.clustering.canopy.CanopyDriver.buildClusters(CanopyDriver.java:167)
    at org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:114)
    at org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:91)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.mahout.clustering.canopy.CanopyDriver.main(CanopyDriver.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
    at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:184)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)





Mime
View raw message