giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: Exception with out-of-core enabled
Date Tue, 11 Feb 2014 11:35:27 GMT
Is there an approximate date when this new DiskBackedPartitionStore will 
become available? I'd be very interested to test it.

--sebastian

On 02/10/2014 07:33 PM, Claudio Martella wrote:
> Hi Sebastian,
>
> this is indeed a bug. Apparently some updates to the PartitionStore API
> introduced a regression. Armando has worked on a re-design of the
> DiskBackedPartitionStore, which we will be uploaded for a review as soon as
> we finish testing it.
>
> Stay tuned.
>
>
> On Mon, Feb 10, 2014 at 6:42 PM, Sebastian Schelter <ssc@apache.org> wrote:
>
>> Hi,
>>
>> I'm trying to process a huge graph on our cluster and therefore enabled
>> out-of-core. Unfortunately, I run into a NullPointerException during the
>> input phase.
>>
>> What can I do to fix this? I'm using the following settings:
>>
>> giraph.oneToAllMsgSending=true
>> giraph.isStaticGraph=true
>> giraph.numComputeThreads=15
>> giraph.numInputThreads=15
>> giraph.numOutputThreads=15
>> giraph.maxNumberOfSupersteps=30
>> giraph.useOutOfCoreGraph=true
>>
>> Here's the log for the exception:
>>
>>
>> java.lang.IllegalStateException: run: Caught an unrecoverable exception
>> waitFor: ExecutionException occurred while waiting for
>> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6e172129
>>          at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>>          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>          at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>          at java.security.AccessController.doPrivileged(Native Method)
>>          at javax.security.auth.Subject.doAs(Subject.java:415)
>>          at org.apache.hadoop.security.UserGroupInformation.doAs(
>> UserGroupInformation.java:1121)
>>          at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
>> occurred while waiting for org.apache.giraph.utils.ProgressableUtils$
>> FutureWaitable@6e172129
>>          at org.apache.giraph.utils.ProgressableUtils.waitFor(
>> ProgressableUtils.java:181)
>>          at org.apache.giraph.utils.ProgressableUtils.waitForever(
>> ProgressableUtils.java:139)
>>          at org.apache.giraph.utils.ProgressableUtils.waitForever(
>> ProgressableUtils.java:124)
>>          at org.apache.giraph.utils.ProgressableUtils.getFutureResult(
>> ProgressableUtils.java:87)
>>          at org.apache.giraph.utils.ProgressableUtils.
>> getResultsWithNCallables(ProgressableUtils.java:221)
>>          at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(
>> BspServiceWorker.java:281)
>>          at org.apache.giraph.worker.BspServiceWorker.loadVertices(
>> BspServiceWorker.java:325)
>>          at org.apache.giraph.worker.BspServiceWorker.setup(
>> BspServiceWorker.java:506)
>>          at org.apache.giraph.graph.GraphTaskManager.execute(
>> GraphTaskManager.java:244)
>>          at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
>>          ... 7 more
>> Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
>> getOrCreatePartition: cannot retrieve partition 14
>>          at java.util.concurrent.FutureTask$Sync.innerGet(
>> FutureTask.java:262)
>>          at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>>          at org.apache.giraph.utils.ProgressableUtils$
>> FutureWaitable.waitFor(ProgressableUtils.java:300)
>>          at org.apache.giraph.utils.ProgressableUtils.waitFor(
>> ProgressableUtils.java:173)
>>          ... 16 more
>> Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot
>> retrieve partition 14
>>          at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:243)
>>          at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
>> doRequest(SendWorkerVerticesRequest.java:110)
>>          at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces
>> sor.doRequest(NettyWorkerClientRequestProcessor.java:482)
>>          at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces
>> sor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276)
>>          at org.apache.giraph.worker.VertexInputSplitsCallable.
>> readInputSplit(VertexInputSplitsCallable.java:172)
>>          at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(
>> InputSplitsCallable.java:267)
>>          at org.apache.giraph.worker.InputSplitsCallable.call(
>> InputSplitsCallable.java:211)
>>          at org.apache.giraph.worker.InputSplitsCallable.call(
>> InputSplitsCallable.java:60)
>>          at org.apache.giraph.utils.LogStacktraceCallable.call(
>> LogStacktraceCallable.java:51)
>>          at java.util.concurrent.FutureTask$Sync.innerRun(
>> FutureTask.java:334)
>>          at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>          at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1110)
>>          at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:603)
>>          at java.lang.Thread.run(Thread.java:722)
>> Caused by: java.util.concurrent.ExecutionException:
>> java.lang.NullPointerException
>>          at java.util.concurrent.FutureTask$Sync.innerGet(
>> FutureTask.java:252)
>>          at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>          at org.apache.giraph.partition.DiskBackedPartitionStore.
>> getOrCreatePartition(DiskBackedPartitionStore.java:228)
>>          ... 13 more
>> Caused by: java.lang.NullPointerException
>>          at org.apache.giraph.partition.DiskBackedPartitionStore$
>> GetPartition.call(DiskBackedPartitionStore.java:692)
>>          at org.apache.giraph.partition.DiskBackedPartitionStore$
>> GetPartition.call(DiskBackedPartitionStore.java:658)
>>          at java.util.concurrent.FutureTask$Sync.innerRun(
>> FutureTask.java:334)
>>          at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>          at org.apache.giraph.partition.DiskBackedPartitionStore$
>> DirectExecutorService.execute(DiskBackedPartitionStore.java:972)
>>          at java.util.concurrent.AbstractExecutorService.submit(
>> AbstractExecutorService.java:132)
>>          ... 14 more
>>
>
>
>


Mime
View raw message