giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastian Schelter <...@apache.org>
Subject Re: GIRAPH-825 and GIRAPH-840
Date Thu, 20 Feb 2014 10:09:54 GMT
Hi Armando & Claudio,

I tested your latest patch, unfortunately it still doesn't work for me.

I applied the patch to the current trunk, copied my algorithm code and 
ran it on the cluster. If it runs in memory, everything works well. If I 
enable out-of-core I run into the deadlock as before.

Here's the command I used:

hadoop jar 
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.hyperball.HyperBall --vertexInputFormat 
org.apache.giraph.examples.hyperball.HyperBallTextInputFormat 
--vertexInputPath hdfs:///ssc/grades/data/twitter-negative/ 
--vertexOutputFormat 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat --outputPath 
hdfs:///ssc/tmp-123/ --combiner 
org.apache.giraph.examples.hyperball.HyperLogLogCombiner --outEdges 
org.apache.giraph.edge.LongNullArrayEdges --workers 24 --customArguments 
giraph.oneToAllMsgSending=true,giraph.isStaticGraph=true,giraph.numComputeThreads=15,giraph.numInputThreads=15,giraph.numOutputThreads=15,giraph.maxNumberOfSupersteps=30,giraph.useOutOfCoreGraph=true,giraph.maxPartitionsInMemory=20

Best,
Sebastian

Here's a stracktrace of one workers:

2014-02-20 11:05:17
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f98f0096000 nid=0x78cb 
waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"netty-server-worker-15" prio=10 tid=0x00007f9910025000 nid=0x77b3 
runnable [0x00007f9914aa1000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000adfc3140> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000adfc3218> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000adfc3040> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-14" prio=10 tid=0x00007f9910023000 nid=0x77b2 
runnable [0x00007f9914ba2000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae3a0680> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae22fae8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae3a0590> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-13" prio=10 tid=0x00007f9910021000 nid=0x77b1 
runnable [0x00007f9914ca3000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae474e30> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae4780c8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae474d40> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-12" prio=10 tid=0x00007f991001f800 nid=0x77b0 
runnable [0x00007f9914da4000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000adfee950> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000adfeec20> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000adfee860> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-11" prio=10 tid=0x00007f991001d800 nid=0x77af 
runnable [0x00007f9914ea5000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae0fc9b0> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae0fcc80> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae0fc8c0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-10" prio=10 tid=0x00007f991001c000 nid=0x77ae 
waiting for monitor entry [0x00007f9914fa6000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-9" prio=10 tid=0x00007f991001a800 nid=0x77ad 
runnable [0x00007f99150a7000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000adf80e78> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000adf81148> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000adf80d88> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-7" prio=10 tid=0x00007f98f8015800 nid=0x77ac waiting 
on condition [0x00007f99151a8000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae444d38> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-8" prio=10 tid=0x00007f9910018800 nid=0x77ab 
runnable [0x00007f99152a9000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae0fd200> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae0fd2d8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae0fd100> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-7" prio=10 tid=0x00007f9910016800 nid=0x77aa 
waiting for monitor entry [0x00007f99153aa000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-6" prio=10 tid=0x00007f9928452800 nid=0x77a9 waiting 
on condition [0x00007f99154ab000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae4466a0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-5" prio=10 tid=0x00007f99006d6000 nid=0x77a8 waiting 
on condition [0x00007f99155ac000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae44df48> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-6" prio=10 tid=0x00007f9910014800 nid=0x77a7 
waiting for monitor entry [0x00007f99156ad000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-4" prio=10 tid=0x00007f9908661800 nid=0x77a6 waiting 
on condition [0x00007f99157ae000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae4287a0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-3" prio=10 tid=0x00007f98f0005800 nid=0x77a5 waiting 
on condition [0x00007f99158af000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae452518> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-5" prio=10 tid=0x00007f9910012800 nid=0x77a4 
waiting for monitor entry [0x00007f99159b0000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-4" prio=10 tid=0x00007f9910010800 nid=0x77a3 
runnable [0x00007f9915ab1000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000adfadba0> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000adfb86d8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000adfadab0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-3" prio=10 tid=0x00007f991000e800 nid=0x77a2 
runnable [0x00007f9915bb1000]
    java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(Native Method)
	at java.io.FileInputStream.read(FileInputStream.java:242)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
	- locked <0x00000005be8e5778> (a java.io.BufferedInputStream)
	at java.io.DataInputStream.readFully(DataInputStream.java:195)
	at java.io.DataInputStream.readLong(DataInputStream.java:416)
	at 
org.apache.giraph.edge.LongNullArrayEdges.readFields(LongNullArrayEdges.java:161)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.readOutEdges(DiskBackedPartitionStore.java:387)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.loadPartition(DiskBackedPartitionStore.java:441)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getPartition(DiskBackedPartitionStore.java:771)
	- locked <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:238)
	- locked <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-2" prio=10 tid=0x0000000000eb2800 nid=0x77a1 waiting 
on condition [0x00007f9915cb3000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae453e70> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-1" prio=10 tid=0x00007f992810c000 nid=0x77a0 waiting 
on condition [0x00007f9915db4000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae45c728> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-2" prio=10 tid=0x00007f991000c000 nid=0x779f 
waiting for monitor entry [0x00007f9915eb5000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-1" prio=10 tid=0x00007f991000a000 nid=0x779e 
waiting for monitor entry [0x00007f9915fb6000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-7" prio=10 tid=0x00007f9928109800 nid=0x779d waiting 
on condition [0x00007f99160b7000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae19d018> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-6" prio=10 tid=0x00007f98e8003800 nid=0x779c waiting 
on condition [0x00007f99161b8000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae1ac6e0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-5" prio=10 tid=0x00007f991800b800 nid=0x779b waiting 
on condition [0x00007f99162b9000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae2367a8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-4" prio=10 tid=0x00007f9900017000 nid=0x779a waiting 
on condition [0x00007f99163ba000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae1adfe8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-3" prio=10 tid=0x00007f9928107800 nid=0x7799 waiting 
on condition [0x00007f99164bb000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae1af8f0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-2" prio=10 tid=0x00007f98e8002000 nid=0x7798 waiting 
on condition [0x00007f99165bc000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae231bf8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-1" prio=10 tid=0x00007f9918002800 nid=0x7797 waiting 
on condition [0x00007f99166bd000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae233500> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-exec-0" prio=10 tid=0x00007f9900015000 nid=0x7796 waiting 
on condition [0x00007f99167be000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae234e08> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-worker-3" prio=10 tid=0x00007f9928036800 nid=0x7795 
runnable [0x00007f99168bf000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae3b2e10> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae22fc08> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae3b2d20> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-worker-2" prio=10 tid=0x00007f9928035000 nid=0x7794 
runnable [0x00007f99169c0000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae24cd00> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae24ff20> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae24cc00> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-worker-1" prio=10 tid=0x00007f9928034000 nid=0x7793 
runnable [0x00007f9916ac1000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae19c400> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae460d48> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae19c310> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-client-worker-0" prio=10 tid=0x00007f9928039800 nid=0x7792 
runnable [0x00007f9916bc2000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae2d0238> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae2d0258> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae2d01f0> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-exec-0" prio=10 tid=0x00007f98f800a000 nid=0x7791 waiting 
on condition [0x00007f9916cc3000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae4577a8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.takeTask(SingleThreadEventExecutor.java:219)
	at 
io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:34)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-worker-0" prio=10 tid=0x00007f9910009800 nid=0x7790 
waiting for monitor entry [0x00007f9916ec5000]
    java.lang.Thread.State: BLOCKED (on object monitor)
	at 
org.apache.giraph.partition.DiskBackedPartitionStore.getOrCreatePartition(DiskBackedPartitionStore.java:234)
	- waiting to lock <0x00000000ae3c8458> (a 
org.apache.giraph.partition.DiskBackedPartitionStore$MetaPartition)
	at 
org.apache.giraph.comm.requests.SendWorkerVerticesRequest.doRequest(SendWorkerVerticesRequest.java:114)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:60)
	at 
org.apache.giraph.comm.netty.handler.WorkerRequestServerHandler.processRequest(WorkerRequestServerHandler.java:36)
	at 
org.apache.giraph.comm.netty.handler.RequestServerHandler.channelRead(RequestServerHandler.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324)
	at 
org.apache.giraph.comm.netty.handler.RequestDecoder.channelRead(RequestDecoder.java:103)
	at 
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338)
	at 
io.netty.channel.DefaultChannelHandlerContext.access$700(DefaultChannelHandlerContext.java:29)
	at 
io.netty.channel.DefaultChannelHandlerContext$8.run(DefaultChannelHandlerContext.java:329)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:354)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:353)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"Thread-12" prio=10 tid=0x00007f9928a74800 nid=0x778c waiting on 
condition [0x00007f9916dc4000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at 
org.apache.giraph.worker.WorkerProgressWriter$1.run(WorkerProgressWriter.java:55)
	at java.lang.Thread.run(Thread.java:722)

"netty-server-boss-0" prio=10 tid=0x00007f9928a73000 nid=0x778a runnable 
[0x00007f9916fc6000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae3e94c8> (a 
io.netty.channel.nio.SelectedSelectionKeySet)
	- locked <0x00000000ae3ea7d0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae3e93c8> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:596)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:306)
	at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
	at java.lang.Thread.run(Thread.java:722)

"main-EventThread" daemon prio=10 tid=0x00007f99287f4000 nid=0x777e 
waiting on condition [0x00007f99170c7000]
    java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae3d6f90> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)

"main-SendThread(cloud-18.dima.tu-berlin.de:22181)" daemon prio=10 
tid=0x00007f99287d5800 nid=0x777d runnable [0x00007f99172c9000]
    java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x00000000ae3d6aa0> (a sun.nio.ch.Util$2)
	- locked <0x00000000ae3d6a90> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000000ae3d6830> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)

"LeaseChecker" daemon prio=10 tid=0x00007f99287cb800 nid=0x777c waiting 
on condition [0x00007f99171c8000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1379)
	at java.lang.Thread.run(Thread.java:722)

"metrics-meter-tick-thread-2" daemon prio=10 tid=0x00007f99287c0000 
nid=0x777a waiting on condition [0x00007f99173ca000]
    java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae3a2fb0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
	at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
	at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

"metrics-meter-tick-thread-1" daemon prio=10 tid=0x00007f99287be000 
nid=0x7779 waiting on condition [0x00007f99174cb000]
    java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ae3a2fb0> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
	at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
	at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

"communication thread" daemon prio=10 tid=0x00007f9928699800 nid=0x7771 
waiting on condition [0x00007f99175cc000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:654)
	at java.lang.Thread.run(Thread.java:722)

"Timer for 'MapTask' metrics system" daemon prio=10 
tid=0x00007f9928673000 nid=0x776f in Object.wait() [0x00007f99179df000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ae393a08> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x00000000ae393a08> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"Thread for syncLogs" daemon prio=10 tid=0x00007f99284e6800 nid=0x776e 
waiting on condition [0x00007f9917ae0000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.hadoop.mapred.Child$3.run(Child.java:139)

"IPC Client (47) connection to /127.0.0.1:36307 from 
job_201402201052_0002" daemon prio=10 tid=0x00007f99284e0800 nid=0x776d 
in Object.wait() [0x00007f9917be1000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ae325778> (a 
org.apache.hadoop.ipc.Client$Connection)
	at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:706)
	- locked <0x00000000ae325778> (a org.apache.hadoop.ipc.Client$Connection)
	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:748)

"Service Thread" daemon prio=10 tid=0x00007f9928105000 nid=0x776b 
runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f9928102800 nid=0x776a 
waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f99280ff800 nid=0x7769 
waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f99280fd800 nid=0x7768 
runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f99280b0800 nid=0x7767 in 
Object.wait() [0x00007f991d50c000]
    java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
	- locked <0x00000000ae4620d8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

"Reference Handler" daemon prio=10 tid=0x00007f99280ae800 nid=0x7766 in 
Object.wait() [0x00007f991d60d000]
    java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:503)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
	- locked <0x00000000ae461d20> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00007f992800b000 nid=0x7757 in Object.wait() 
[0x00007f993111b000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000ae3b3740> (a 
java.util.concurrent.ConcurrentHashMap)
	at 
org.apache.giraph.comm.netty.NettyClient.waitSomeRequests(NettyClient.java:710)
	- locked <0x00000000ae3b3740> (a java.util.concurrent.ConcurrentHashMap)
	at 
org.apache.giraph.comm.netty.NettyClient.waitAllRequests(NettyClient.java:685)
	at 
org.apache.giraph.comm.netty.NettyWorkerClient.waitAllRequests(NettyWorkerClient.java:148)
	at 
org.apache.giraph.worker.BspServiceWorker.loadInputSplits(BspServiceWorker.java:291)
	at 
org.apache.giraph.worker.BspServiceWorker.loadVertices(BspServiceWorker.java:328)
	at 
org.apache.giraph.worker.BspServiceWorker.setup(BspServiceWorker.java:509)
	at 
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:262)
	at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:91)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)

"VM Thread" prio=10 tid=0x00007f99280a7000 nid=0x7765 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f9928018800 
nid=0x7758 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f992801a800 
nid=0x7759 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f992801c800 
nid=0x775a runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f992801e800 
nid=0x775b runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f9928020000 
nid=0x775c runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f9928022000 
nid=0x775d runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f9928024000 
nid=0x775e runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f9928025800 
nid=0x775f runnable

"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f9928027800 
nid=0x7760 runnable

"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f9928029800 
nid=0x7761 runnable

"GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f992802b000 
nid=0x7762 runnable

"GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f992802d000 
nid=0x7763 runnable

"GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f992802f000 
nid=0x7764 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f9928117800 nid=0x776c 
waiting on condition

JNI global references: 414


On 02/16/2014 12:01 AM, Armando Miraglia wrote:
> Hi Sebastian.
>
> I gave a quick look to the source code of DiskPartition and I realised
> I forgot to correctly manage the concurrent management of one variable.
> The patch I am sending you now should fix this problem. However, I am
> not sure whether this was the reason for your issue, so it is still
> possible that your issue is there.
>
> If you could test it, it would be great.
>
> Cheers,
> A.
>


Mime
View raw message