giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Exception with out-of-core enabled
Date Mon, 10 Feb 2014 18:33:35 GMT
Hi Sebastian,

this is indeed a bug. Apparently some updates to the PartitionStore API
introduced a regression. Armando has worked on a re-design of the
DiskBackedPartitionStore, which we will be uploaded for a review as soon as
we finish testing it.

Stay tuned.


On Mon, Feb 10, 2014 at 6:42 PM, Sebastian Schelter <ssc@apache.org> wrote:

> Hi,
>
> I'm trying to process a huge graph on our cluster and therefore enabled
> out-of-core. Unfortunately, I run into a NullPointerException during the
> input phase.
>
> What can I do to fix this? I'm using the following settings:
>
> giraph.oneToAllMsgSending=true
> giraph.isStaticGraph=true
> giraph.numComputeThreads=15
> giraph.numInputThreads=15
> giraph.numOutputThreads=15
> giraph.maxNumberOfSupersteps=30
> giraph.useOutOfCoreGraph=true
>
> Here's the log for the exception:
>
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception
> waitFor: ExecutionException occurred while waiting for
> org.apache.giraph.utils.ProgressableUtils$FutureWaitable@6e172129
>         at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException
> occurred while waiting for org.apache.giraph.utils.ProgressableUtils$
> FutureWaitable@6e172129
>         at org.apache.giraph.utils.ProgressableUtils.waitFor(
> ProgressableUtils.java:181)
>         at org.apache.giraph.utils.ProgressableUtils.waitForever(
> ProgressableUtils.java:139)
>         at org.apache.giraph.utils.ProgressableUtils.waitForever(
> ProgressableUtils.java:124)
>         at org.apache.giraph.utils.ProgressableUtils.getFutureResult(
> ProgressableUtils.java:87)
>         at org.apache.giraph.utils.ProgressableUtils.
> getResultsWithNCallables(ProgressableUtils.java:221)
>         at org.apache.giraph.worker.BspServiceWorker.loadInputSplits(
> BspServiceWorker.java:281)
>         at org.apache.giraph.worker.BspServiceWorker.loadVertices(
> BspServiceWorker.java:325)
>         at org.apache.giraph.worker.BspServiceWorker.setup(
> BspServiceWorker.java:506)
>         at org.apache.giraph.graph.GraphTaskManager.execute(
> GraphTaskManager.java:244)
>         at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
>         ... 7 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException:
> getOrCreatePartition: cannot retrieve partition 14
>         at java.util.concurrent.FutureTask$Sync.innerGet(
> FutureTask.java:262)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>         at org.apache.giraph.utils.ProgressableUtils$
> FutureWaitable.waitFor(ProgressableUtils.java:300)
>         at org.apache.giraph.utils.ProgressableUtils.waitFor(
> ProgressableUtils.java:173)
>         ... 16 more
> Caused by: java.lang.IllegalStateException: getOrCreatePartition: cannot
> retrieve partition 14
>         at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:243)
>         at org.apache.giraph.comm.requests.SendWorkerVerticesRequest.
> doRequest(SendWorkerVerticesRequest.java:110)
>         at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces
> sor.doRequest(NettyWorkerClientRequestProcessor.java:482)
>         at org.apache.giraph.comm.netty.NettyWorkerClientRequestProces
> sor.sendVertexRequest(NettyWorkerClientRequestProcessor.java:276)
>         at org.apache.giraph.worker.VertexInputSplitsCallable.
> readInputSplit(VertexInputSplitsCallable.java:172)
>         at org.apache.giraph.worker.InputSplitsCallable.loadInputSplit(
> InputSplitsCallable.java:267)
>         at org.apache.giraph.worker.InputSplitsCallable.call(
> InputSplitsCallable.java:211)
>         at org.apache.giraph.worker.InputSplitsCallable.call(
> InputSplitsCallable.java:60)
>         at org.apache.giraph.utils.LogStacktraceCallable.call(
> LogStacktraceCallable.java:51)
>         at java.util.concurrent.FutureTask$Sync.innerRun(
> FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
>         at java.util.concurrent.FutureTask$Sync.innerGet(
> FutureTask.java:252)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>         at org.apache.giraph.partition.DiskBackedPartitionStore.
> getOrCreatePartition(DiskBackedPartitionStore.java:228)
>         ... 13 more
> Caused by: java.lang.NullPointerException
>         at org.apache.giraph.partition.DiskBackedPartitionStore$
> GetPartition.call(DiskBackedPartitionStore.java:692)
>         at org.apache.giraph.partition.DiskBackedPartitionStore$
> GetPartition.call(DiskBackedPartitionStore.java:658)
>         at java.util.concurrent.FutureTask$Sync.innerRun(
> FutureTask.java:334)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>         at org.apache.giraph.partition.DiskBackedPartitionStore$
> DirectExecutorService.execute(DiskBackedPartitionStore.java:972)
>         at java.util.concurrent.AbstractExecutorService.submit(
> AbstractExecutorService.java:132)
>         ... 14 more
>



-- 
   Claudio Martella

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message