giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hassan Eslami <hsn.esl...@gmail.com>
Subject Re: Out of core thread exception : java concurrent exception
Date Sat, 14 May 2016 22:24:57 GMT
Hi Ramesh!

Thanks for bringing this up, and thanks for trying out the new out-of-core
mechanism. The new out-of-core mechanism has not been integrated with
checkpointing yet. This is part of an ongoing project, and we should have
the integration within a few weeks. In the meantime, you can try
out-of-core without checkpointing enabled.

Best,
Hassan

On Saturday, May 14, 2016, Ramesh Krishnan <ramesh.154089@gmail.com> wrote:

> PFA the correct logs for the concurrent exception
>
> 2016-05-14 19:10:55,733 ERROR [ooc-io-0] org.apache.giraph.utils.LogStacktraceCallable:
Execution of callable failed
> java.lang.RuntimeException: java.io.EOFException
> 	at org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:76)
> 	at org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:30)
> 	at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:392)
> 	at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:47)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.readOutEdges(DiskBackedPartitionStore.java:286)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadInMemoryPartitionData(DiskBackedPartitionStore.java:329)
> 	at org.apache.giraph.ooc.data.OutOfCoreDataManager.loadPartitionData(OutOfCoreDataManager.java:195)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.loadPartitionData(DiskBackedPartitionStore.java:360)
> 	at org.apache.giraph.ooc.io.LoadPartitionIOCommand.execute(LoadPartitionIOCommand.java:64)
> 	at org.apache.giraph.ooc.OutOfCoreIOCallable.call(OutOfCoreIOCallable.java:72)
> 	... 6 more
> 2016-05-14 19:10:55,737 INFO [ooc-io-0] org.apache.giraph.ooc.OutOfCoreIOCallableFactory:
afterExecute: an out-of-core thread terminated unexpectedly with java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.io.EOFException
> 2016-05-14 19:10:55,739 INFO [checkpoint-vertices-7] org.apache.giraph.ooc.FixedOutOfCoreEngine:
getNextPartition: waiting until a partition becomes available!
> 2016-05-14 19:10:56,426 ERROR [checkpoint-vertices-6] org.apache.giraph.utils.LogStacktraceCallable:
Execution of callable failed
> java.lang.RuntimeException: Job Failed due to a failure in an out-of-core IO thread
> 	at org.apache.giraph.ooc.FixedOutOfCoreEngine.getNextPartition(FixedOutOfCoreEngine.java:81)
> 	at org.apache.giraph.ooc.data.DiskBackedPartitionStore.getNextPartition(DiskBackedPartitionStore.java:187)
> 	at org.apache.giraph.worker.BspServiceWorker$3$1.call(BspServiceWorker.java:1398)
> 	at org.apache.giraph.worker.BspServiceWorker$3$1.call(BspServiceWorker.java:1392)
> 	at org.apache.giraph.utils.LogStacktraceCallable.call(LogStacktraceCallable.java:51)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
>
>
>
> On Sun, May 15, 2016 at 1:02 AM, Ramesh Krishnan <ramesh.154089@gmail.com
> <javascript:_e(%7B%7D,'cvml','ramesh.154089@gmail.com');>> wrote:
>
>>
>> Hi Team,
>>
>> I have the latest build of giraph running on a 5 node cluster. When i try
>> to use OutofCore Graph option for a huge data set like 600Milion edges i am
>> running into
>> the following exception. Please find below the script being executed and
>> the exception logs. I have tried all possible ways and could not avoid this
>> issue , i am really in need of your help.
>>
>> *Script:*hadoop jar
>> /usr/local/giraph.back.1.2.0/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.7.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner -Dmapreduce.task.timeout=12000000
>> -Dmapred.job.tracker=ip-172-31-42-220.eu-west-1.compute.internal:8021
>> -Dmapreduce.map.memory.mb=23480 -Dmapreduce.map.java.opts=-Xmx22480m
>> org.apache.giraph.examples.ConnectedComponentsComputation   -vif
>> org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /test/input_10M
>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /test/ouput_10M -w 5 -ca
>> giraph.userPartitionCount=150,giraph.SplitMasterWorker=true,giraph.isStaticGraph=true,giraph.maxPartitionsInMemory=10,mapred.map.max.attempts=2,giraph.maxMessagesInMemory=100,giraph.numOutputThreads=10,giraph.useOutOfCoreMessages=true,giraph.numOutputThreads=4,giraph.numInputThreads=4,giraph.useOutOfCoreGraph=true,giraph.cleanupCheckpointsAfterSuccess=true,giraph.checkpointFrequency=1
>>
>>
>>
>>
>> *Exception:hadoop jar
>> /usr/local/giraph.back.1.2.0/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.7.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner -Dmapreduce.task.timeout=12000000
>> -Dmapred.job.tracker=ip-172-31-42-220.eu-west-1.compute.internal:8021
>> -Dmapreduce.map.memory.mb=23480 -Dmapreduce.map.java.opts=-Xmx22480m
>> org.apache.giraph.examples.ConnectedComponentsComputation   -vif
>> org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip /test/input_10M
>> -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /test/ouput_10M -w 5 -ca
>> giraph.userPartitionCount=150,giraph.SplitMasterWorker=true,giraph.isStaticGraph=true,giraph.maxPartitionsInMemory=10,mapred.map.max.attempts=2,giraph.maxMessagesInMemory=100,giraph.numOutputThreads=10,giraph.useOutOfCoreMessages=true,giraph.numOutputThreads=4,giraph.numInputThreads=4,giraph.useOutOfCoreGraph=true,giraph.cleanupCheckpointsAfterSuccess=true,giraph.checkpointFrequency=1*
>>
>> *thanks*
>>
>> *Ramesh*
>>
>>
>

Mime
View raw message