mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From syed kather <in.ab...@gmail.com>
Subject Heap Space Problem while running in cluster in map reduce
Date Wed, 03 Oct 2012 19:14:48 GMT
Team,
  When i am trying to run KMean clustering i had found it is throwing
  Java heap space
         at
org.apache.mahout.math.map.OpenIntDoubleHashMap.rehash(OpenIntDoubleHashMap.java:434)
         at
org.apache.mahout.math.map.OpenIntDoubleHashMap.put(OpenIntDoubleHashMap.java:387)
         at
org.apache.mahout.math.RandomAccessSparseVector.setQuick(RandomAccessSparseVector.java:139)
         at
org.apache.mahout.math.VectorWritable.readFields(VectorWritable.java:118)
         at
org.apache.mahout.clustering.ClusterObservations.readFields(ClusterObservations.java:59)
         at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
         at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
         at
org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
         at
org.apache.hadoop.mapreduce.ReduceContext$ValueIterator.next(ReduceContext.java:163)
         at
org.apache.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:31)
         at
org.apache.mahout.clustering.kmeans.KMeansCombiner.reduce(KMeansCombiner.java:25)
         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
         at
org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1502)
         at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.doInMemMerge(ReduceTask.java:2768)
        at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2706)

Can i know what may be the reason.

    I have 5 Node cluster
    Master      4 Core with 16GB RAM
    salve1       4 Core with  8GB RAM
    salve2       4 Core with  8GB RAM
    salve3       4 Core with  8GB RAM
    salve4       4 Core with  8GB RAM

Let me know if there is any optimization is required for this

Advance Thanks
            Thanks and Regards,
        S SYED ABDUL KATHER

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message