giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Kumar <toga...@gmail.com>
Subject giraph memory usage info
Date Mon, 14 Apr 2014 13:32:17 GMT
Hello,

I am trying to run triangle count program in giraph using a hadoop cluster
of 6 machines each having 16 gb of ram. My data set is of  size 1 gb which
is of json format.

The issue is I am getting  OutOfMemoryError  while trying to run this. For
data which is up to 25 thousand line of code it runs fine.I have updated

Hadoop-env.sh with export HADOOP_HEAPSIZE=2000 and mapred-site.xml with
<name>mapred.child.java.opts</name>
   <value>-Xmx3000m -XX:+UseConcMarkSweepGC</value>
and the number of map instance is max 3  so for each machine  will occupy
13 gb

But even after doing all this things I am getting the below exception,

Error from attempt_201404140400_0001_m_000011_0:
java.lang.IllegalStateException: run: Caught an unrecoverable exception
waitFor: ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@3e3a2536
        at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@3e3a2536
        at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
        at
org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:111)
        at
org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:73)
        at
org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:192)
        at
org.apache.giraph.graph.GraphTaskManager.processGraphPartitions(GraphTaskManager.java:753)
        at
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:273)
        at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
        ... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
        at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
        at java.util.concurrent.FutureTask.get(FutureTask.java:91)
        at
org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:271)
        at
org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
        ... 13 more
Caused by: java.lang.OutOfMemoryError: Java heap space

Thanks,

Arun

Mime
View raw message