giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hassan Eslami <hsn.esl...@gmail.com>
Subject Re: Out of core graph option failing with buffer underflow exception
Date Tue, 26 Jul 2016 16:24:46 GMT
Hi Ramesh,

First of all, it seems that you don't want the unsafe-serialization for
some reason, and that makes the out-of-core to be very slow.
(De)Serialization is one of the performance critical components in the new
out-of-core design.

Second, you are using the out-of-core flags based on the documentation on
the website. Most of those flags don't exist anymore, and the documentation
on the website is stale. I've had a couple of diffs to clean up the
out-of-core flags as well as the documentation on the website, but the team
didn't let the diffs to go in the code base. I'm not sure if those diffs
would be part of the code base anytime soon. Without those cleanups and
changes, the out-of-core can be enabled with some other set of flags, which
are pretty complicated to understand (and use)! At this point, I simply
*discourage* using the new out-of-core design. You can simply use the 1.1
release and use the old out-of-core if that works for you.

Hassan

On Mon, Jul 25, 2016 at 9:35 PM, Ramesh Krishnan <ramesh.154089@gmail.com>
wrote:

> Hi Team,
> I have been trying out of core graph option with 1.2 branch of giraph . I
> have been
> continously recieving the buffer underflow exception exactly when the
> superstep1 starts. Any pointers would be really helpful here ,
>
> Logs:
>
> 2016-07-25 16:08:20,664 ERROR [ooc-io-0] org.apache.giraph.utils.LogStacktraceCallable:
Execution of callable failed
> java.lang.RuntimeException: call: execution of IO command LoadPartitionIOCommand: (partitionId
= 258, superstep = 1) failed!
> 	at org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:115)
> 	at org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:36)
> 	at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:67)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow.
> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> 	at com.esotericsoftware.kryo.io.UnsafeInput.readInt(UnsafeInput.java:82)
> 	at com.esotericsoftware.kryo.io.KryoDataInput.readInt(KryoDataInput.java:87)
> 	at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:47)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.readOutEdges(DiskBackedPartitionStore.java:245)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadInMemoryPartitionData(DiskBackedPartitionStore.java:278)
> 	at org.apache.giraph.ooc.data.DiskBackedDataStore.loadPartitionDataProxy(DiskBackedDataStore.java:233)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadPartitionData(DiskBackedPartitionStore.java:311)
> 	at org.apache.giraph.ooc.command.LoadPartitionIOCommand.execute(LoadPartitionIOCommand.java:66)
> 	at org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:99)
> 	... 6 more
>
> Command:
> hadoop jar /usr/local/giraph-1.2/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar
 org.apache.giraph.GiraphRunner -Dmapreduce.task.timeout=12000000 -Dmapred.job.tracker=ip-172-31-42-220.eu-west-1.compute.internal:8021
-Dmapreduce.map.memory.mb=43480 -Dmapreduce.map.java.opts=-Xmx42480m org.apache.giraph.examples.ConnectedComponentsComputation
  -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /VUID/input_2B -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-op /VUID/ouput_2B -w 4 -ca giraph.userPartitionCount=400,giraph.SplitMasterWorker=true,giraph.isStaticGraph=true,giraph.maxPartitionsInMemory=10,mapred.map.max.attempts=2,giraph.maxMessagesInMemory=100,giraph.numOutputThreads=1,giraph.useOutOfCoreMessages=true,giraph.numInputThreads=1,giraph.useOutOfCoreGraph=true,giraph.numComputeThreads=1,giraph.messagesBufferSize=8192000,giraph.partitionsDirectory=_bs,giraph.useUnsafeSerialization=false
>
>
> Thanks,
>
> Ramesh
>
>
>

Mime
View raw message