hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tommaso Teofili <tommaso.teof...@gmail.com>
Subject Re: Some Question of Hama Program on Large Dataset.
Date Tue, 14 Jan 2014 13:24:10 GMT
You may also use OffHeapVerticesInfo [1] to store vertices off heap and
avoid consuming too much memory for that.

Regards,
Tommaso

[1] :
https://svn.apache.org/repos/asf/hama/trunk/graph/src/main/java/org/apache/hama/graph/OffHeapVerticesInfo.java

2014/1/14 Chia-Hung Lin <clin4j@googlemail.com>

> For OOM, memory related option `bsp.child.java.opts' can be added to
> hama-site.xml for increasing memory size; or consider reducing memory
> footprint.
>
> Based on the setting at hand, max tasks, i.e. peer, can be configured
> via `bsp.tasks.maximum' with default set to 3.
>
> IIRC, peer process is something like child process in original
> mapreduce where task tracker will fork a child processing algorithm,
> mainly for fault isolation.
>
>
>
>
> On 14 January 2014 17:21, 步青云 <mailliuping@qq.com> wrote:
> > Hello,
> >      I've got some problem about Hama program on large dataset.My hama
> cluster has 3 computers.One is bspmaster,other 2 computers is
> groomserver.When the graph has one million nodes,the pagerank running will
> stop at "Current Superstep Number:0".The error information is as follows:
> >
> >
> > attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 ERROR bsp.BSPTask:
> Shutting down ping service.
> > attempt_201401141543_0015_000000_0: 14/01/14 17:07:55 FATAL
> bsp.GroomServer: Error running child
> > attempt_201401141543_0015_000000_0: java.lang.OutOfMemoryError: Java
> heap space
> > attempt_201401141543_0015_000000_0:     at
> java.util.Arrays.copyOf(Arrays.java:2271)
> > attempt_201401141543_0015_000000_0:     at
> java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
> > attempt_201401141543_0015_000000_0:     at
> java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> > attempt_201401141543_0015_000000_0:     at
> java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
> > attempt_201401141543_0015_000000_0:     at
> java.io.DataOutputStream.write(DataOutputStream.java:107)
> >       Should I add the number of computers of cluster to solve this
> problem?Any other solution?
> >
> >       And I also want to know how many peers a groomserver can run?Is a
> peer correspond to a node in graph mode?If so,I have one million of nodes
> in a graph and if one groomserver run 500 peers,so I need 2000 groomservers
> correspond to 2000 computers.In my opinion,it's not convenient.
> >      By the way,what's a appropriate value of "bsp.tasks.maxinum" in
> Hama conf ?Is the number of computer of cluster a appropriate value?
> >      Thanks!
> >      Best wishes.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message