giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Agrta Rawat <agrta.ra...@gmail.com>
Subject Re: giraph memory usage info
Date Tue, 15 Apr 2014 08:43:30 GMT
Hi Arun,

 I would suggest you to check two things -

1. Does your code run successfully for some small data?
2. Have you modified hadoop-env.sh and mapred-site.xml file for all 6
clusters?

And it is showing  "Caused by: java.util.concurrent.

ExecutionException: java.lang.OutOfMemoryError: Java heap space"


This means there is something in your code that is being modified and
accessed simultaneously at runtime.

e.g. if you are iterating a hashset and removing or adding contents to it,
it will give such kind of exceptions.


Regards,

Agrta Rawat


On Mon, Apr 14, 2014 at 7:02 PM, Arun Kumar <togarun@gmail.com> wrote:

> Hello,
>
> I am trying to run triangle count program in giraph using a hadoop cluster
> of 6 machines each having 16 gb of ram. My data set is of  size 1 gb which
> is of json format.
>
> The issue is I am getting  OutOfMemoryError  while trying to run this. For
> data which is up to 25 thousand line of code it runs fine.I have updated
>
> Hadoop-env.sh with export HADOOP_HEAPSIZE=2000 and mapred-site.xml with
> <name>mapred.child.java.opts</name>
>    <value>-Xmx3000m -XX:+UseConcMarkSweepGC</value>
> and the number of map instance is max 3  so for each machine  will occupy
> 13 gb
>
> But even after doing all this things I am getting the below exception,
>
> Error from attempt_201404140400_0001_m_000011_0:
> java.lang.IllegalStateException: run: Caught an unrecoverable exception
> waitFor: ExecutionException occurred while waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@3e3a2536
>         at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
> occurred while waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@3e3a2536
>         at
> org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
>         at
> org.apache.giraph.utils.ProgressableUtils.waitForever(ProgressableUtils.java:111)
>         at
> org.apache.giraph.utils.ProgressableUtils.getFutureResult(ProgressableUtils.java:73)
>         at
> org.apache.giraph.utils.ProgressableUtils.getResultsWithNCallables(ProgressableUtils.java:192)
>         at
> org.apache.giraph.graph.GraphTaskManager.processGraphPartitions(GraphTaskManager.java:753)
>         at
> org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:273)
>         at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
>         ... 7 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.OutOfMemoryError: Java heap space
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>         at
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable.waitFor(ProgressableUtils.java:271)
>         at
> org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
>         ... 13 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> Thanks,
>
> Arun
>

Mime
View raw message