giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hassan Eslami <hsn.esl...@gmail.com>
Subject Re: Problems with running page rank using OutOfCore setting
Date Sun, 23 Jul 2017 22:36:12 GMT
Darshan,

Please follow the discussion here
<http://mail-archives.apache.org/mod_mbox/giraph-user/201610.mbox/%3CCAEUY=V-iRV94aj_QkQzF-7tPsaDzadb8VgWVGjiWNCQxjaOBsg@mail.gmail.com%3E>
.

After that issue is resolved, you can take a look at this discussion
<http://mail-archives.apache.org/mod_mbox/giraph-user/201611.mbox/%3CCAH1LQffrK9VwJU%3Dwx7Pi-7TCyowtAKvUMy%2BrrJq5t0M3Q-UgZA%40mail.gmail.com%3E>
on some guidelines to use out-of-core feature.

Best,
Hassan

On Mon, Jul 17, 2017 at 3:44 AM, Darshan Mallenahalli Shankaralingappa <
dshankaralingappa@ntent.com> wrote:

> Hi,
>
> I added -Dgiraph.waitForPerWorkerRequests=true parameter. And I got this
> error.
>
>
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
>         at java.util.ArrayList$SubList.listIterator(ArrayList.java:1095)
>         at java.util.AbstractList.listIterator(AbstractList.java:299)
>         at java.util.ArrayList$SubList.iterator(ArrayList.java:1087)
>         at java.util.AbstractCollection.toArray(AbstractCollection.
> java:180)
>         at java.util.regex.Pattern.split(Pattern.java:1241)
>         at java.util.regex.Pattern.split(Pattern.java:1273)
>         at org.apache.giraph.examples.LongDoubleNullTextInputFormat$
> LongDoubleNullDoubleVertexReader.getCurrentVertex(
> LongDoubleNullTextInputFormat.java:86)
>         at org.apache.giraph.io.internal.WrappedVertexReader.
> getCurrentVertex(WrappedVertexReader.java:90)
>         at org.apache.giraph.worker.VertexInputSplitsCallable.
> readInputSplit(VertexInputSplitsCallable.java:182)
>         at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(
> InputSplitsCallable.java:275)
>         at org.apache.giraph.worker.InputSplitsCallable.call(
> InputSplitsCallable.java:227)
>         at org.apache.giraph.worker.InputSplitsCallable.call(
> InputSplitsCallable.java:60)
>         at org.apache.giraph.utils.LogStacktraceCallable.call(
> LogStacktraceCallable.java:67)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
> So, does this mean that there is no other solution but to increase the
> physical memory?
>
>
> Cheers,
>
> Darshan
>
>
> On 16 Jul 2017, at 23:04, Hassan Eslami <hsn.eslami@gmail.com<mailto:h
> sn.eslami@gmail.com>> wrote:
>
> Hi,
>
> giraph.useOutOfCoreMessages is no longer in use.
>
> The main problem here is that you are using default flow control mechanism
> (NoOpFlowControl), that causes a lot of outstanding/received messages. As a
> consequence, you fill up the memory so fast, and the job would fail for
> various reasons. Please use the following options instead:
>
> -Dgiraph.isStaticGraph=false -Dgiraph.useOutOfCoreGraph=true
> -Dgiraph.waitForPerWorkerRequests=true
>
> Note: the static graph has a known bug with the out-of-core mechanism.
>
> Hope it helps,
> Hassan
>
> On Sun, Jul 16, 2017 at 1:54 PM, Darshan Mallenahalli Shankaralingappa <
> dshankaralingappa@ntent.com<mailto:dshankaralingappa@ntent.com>> wrote:
>
> Hi,
>
> I am trying to run the page rank algorithm using giraph on a 3.5 billion
> node web graph on a relatively smaller Hadoop cluster (6 nodes with 225GB
> RAM total).
> I set the giraph.useOutOfCoreGraph and giraph.useOutOfCoreMessages to true
> and the application killed after some time.
>
> I am running the giraph job using this command:
> yarn jar giraph-examples-1.2.0-for-hadoop-2.6.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner -Dgiraph.yarn.task.heap.mb=58880
> -Dgiraph.isStaticGraph=true -Dgiraph.useOutOfCoreGraph=true
> -Dgiraph.useOutOfCoreMessages=true org.apache.giraph.examples.
> PageRankComputation
> -vif org.apache.giraph.examples.LongDoubleNullTextInputFormat -vip
> /user/darshan/AdjList/ -vof org.apache.giraph.io.formats.
> IdWithValueTextOutputFormat
> -op /user/darshan/giraph_3.5B_ooc/ -w 8 -mc org.apache.giraph.examples.
> RandomWalkVertexMasterCompute
> -wc org.apache.giraph.examples.RandomWalkWorkerContext -ca
> org.apache.giraph.examples.RandomWalkVertex.teleportationProbability=0.15f
> -ca org.apache.giraph.examples.RandomWalkVertex.maxSupersteps=21
>
> Here is a log from the zookeeper:
>
> 2017-07-12 08:08:35,026 WARN [netty-client-worker-1]
> org.apache.giraph.comm.netty.handler.ResponseClientHandler:
> exceptionCaught: Channel failed with remote address <url>/<ip>:30006<
> http://hdpbcn-01.lv.ntent.com/10.100.21.118:30006>
>
> java.lang.ArrayIndexOutOfBoundsException: 1075052547
>        at org.apache.giraph.comm.flow_control.NoOpFlowControl.
> getAckSignalFlag(NoOpFlowControl.java:52)
>        at org.apache.giraph.comm.netty.NettyClient.messageReceived(
> NettyClient.java:796)
>        at org.apache.giraph.comm.netty.handler.ResponseClientHandler.
> channelRead(ResponseClientHandler.java:87)
>        at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
>        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(
> ByteToMessageDecoder.java:153)
>        at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
>        at org.apache.giraph.comm.netty.InboundByteCounter.channelRead(
> InboundByteCounter.java:74)
>        at io.netty.channel.DefaultChannelHandlerContext.
> invokeChannelRead(DefaultChannelHandlerContext.java:338)
>        at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(
> DefaultChannelHandlerContext.java:324)
>        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(
> DefaultChannelPipeline.java:785)
>        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(
> AbstractNioByteChannel.java:126)
>        at io.netty.channel.nio.NioEventLoop.processSelectedKey(
> NioEventLoop.java:485)
>        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(
> NioEventLoop.java:452)
>        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346)
>        at io.netty.util.concurrent.SingleThreadEventExecutor$2.
> run(SingleThreadEventExecutor.java:101)
>        at java.lang.Thread.run(Thread.java:745)
>
>
> I think this issue is related to the messaging stack rather than the
> algorithm.
> If not, can someone please help me with this or at least point me in the
> right direction?
>
> Cheers,
> Darshan
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message