hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Naik <sn...@attributor.com>
Subject Re: Hadoop 18.1 "ls" stalls
Date Thu, 20 Nov 2008 19:31:50 GMT
Thanks Raghu,
*datapoints:*
 - So when I use FSShell client, it gets into retry mode for 
getFilesInfo() call and takes a long time.

  - Also, when do a "ls" operation, it takes secs(4/5) .

  - 1.6 million files and namenode is mostly full with heap(2400M)  
(from ui)


The jstack is



2008-11-20 11:24:27
Full thread dump Java HotSpot(TM) Server VM (10.0-b19 mixed mode):

"Attach Listener" daemon prio=10 tid=0x0a7e6000 nid=0x538a waiting on 
condition [0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"SocketListener0-2" prio=10 tid=0x13336800 nid=0x6bc8 in Object.wait() 
[0x14f67000..0x14f67ed0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x48ae6df0> (a org.mortbay.util.ThreadPool$PoolThread)
    at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
    - locked <0x48ae6df0> (a org.mortbay.util.ThreadPool$PoolThread)

   Locked ownable synchronizers:
    - None

"Trash Emptier" daemon prio=10 tid=0x17070800 nid=0x428 waiting on 
condition [0x14f17000..0x14f171d0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.fs.Trash$Emptier.run(Trash.java:219)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"org.apache.hadoop.dfs.DFSClient$LeaseChecker@159d510" daemon prio=10 
tid=0x17070400 nid=0x427 sleeping[0x14fb9000..0x14fb9150]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:792)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"IPC Server handler 99 on 54310" daemon prio=10 tid=0x17061400 nid=0x41f 
waiting on condition [0x15009000..0x1500a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 98 on 54310" daemon prio=10 tid=0x09956000 nid=0x41e 
waiting on condition [0x1505b000..0x1505b1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 97 on 54310" daemon prio=10 tid=0x08b36400 nid=0x41d 
waiting on condition [0x150ab000..0x150ac150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 96 on 54310" daemon prio=10 tid=0x17418400 nid=0x41c 
waiting on condition [0x150fc000..0x150fced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 95 on 54310" daemon prio=10 tid=0x1705dc00 nid=0x41b 
waiting on condition [0x1514d000..0x1514de50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 94 on 54310" daemon prio=10 tid=0x1705cc00 nid=0x41a 
waiting on condition [0x1519e000..0x1519efd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 93 on 54310" daemon prio=10 tid=0x1705a800 nid=0x419 
waiting on condition [0x151ef000..0x151eff50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 92 on 54310" daemon prio=10 tid=0x17059800 nid=0x418 
waiting on condition [0x15240000..0x152410d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 91 on 54310" daemon prio=10 tid=0x17058400 nid=0x417 
waiting on condition [0x15291000..0x15292050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 90 on 54310" daemon prio=10 tid=0x17057400 nid=0x416 
waiting on condition [0x152e3000..0x152e31d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 89 on 54310" daemon prio=10 tid=0x17056000 nid=0x415 
waiting on condition [0x15333000..0x15334150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 88 on 54310" daemon prio=10 tid=0x17055000 nid=0x414 
waiting on condition [0x15384000..0x15384ed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 87 on 54310" daemon prio=10 tid=0x17053c00 nid=0x413 
waiting on condition [0x153d5000..0x153d5e50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 86 on 54310" daemon prio=10 tid=0x17052800 nid=0x412 
waiting on condition [0x15426000..0x15426fd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 85 on 54310" daemon prio=10 tid=0x17051800 nid=0x411 
waiting on condition [0x15477000..0x15477f50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 84 on 54310" daemon prio=10 tid=0x17050400 nid=0x410 
waiting on condition [0x154c8000..0x154c90d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 83 on 54310" daemon prio=10 tid=0x1704f400 nid=0x40f 
waiting on condition [0x15519000..0x1551a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 82 on 54310" daemon prio=10 tid=0x1704e400 nid=0x40e 
waiting on condition [0x1556b000..0x1556b1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 81 on 54310" daemon prio=10 tid=0x1704d000 nid=0x40d 
waiting on condition [0x155bb000..0x155bc150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 80 on 54310" daemon prio=10 tid=0x1704c000 nid=0x40c 
waiting on condition [0x1560c000..0x1560ced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 79 on 54310" daemon prio=10 tid=0x1704ac00 nid=0x40b 
waiting on condition [0x1565d000..0x1565de50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 78 on 54310" daemon prio=10 tid=0x17049c00 nid=0x40a 
waiting on condition [0x156ae000..0x156aefd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 77 on 54310" daemon prio=10 tid=0x17048800 nid=0x409 
waiting on condition [0x156ff000..0x156fff50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 76 on 54310" daemon prio=10 tid=0x17047800 nid=0x408 
waiting on condition [0x15750000..0x157510d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 75 on 54310" daemon prio=10 tid=0x17046400 nid=0x407 
waiting on condition [0x157a1000..0x157a2050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 74 on 54310" daemon prio=10 tid=0x17045400 nid=0x406 
waiting on condition [0x157f3000..0x157f31d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 73 on 54310" daemon prio=10 tid=0x17044000 nid=0x405 
waiting on condition [0x15843000..0x15844150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 72 on 54310" daemon prio=10 tid=0x17043000 nid=0x404 
waiting on condition [0x15894000..0x15894ed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 71 on 54310" daemon prio=10 tid=0x17041c00 nid=0x403 
waiting on condition [0x158e5000..0x158e5e50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 70 on 54310" daemon prio=10 tid=0x17040c00 nid=0x402 
waiting on condition [0x15936000..0x15936fd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 69 on 54310" daemon prio=10 tid=0x1703f800 nid=0x401 
waiting on condition [0x15987000..0x15987f50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 68 on 54310" daemon prio=10 tid=0x1703e800 nid=0x400 
waiting on condition [0x159d8000..0x159d90d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 67 on 54310" daemon prio=10 tid=0x1703d400 nid=0x3ff 
waiting on condition [0x15a29000..0x15a2a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 66 on 54310" daemon prio=10 tid=0x1703c400 nid=0x3fe 
waiting on condition [0x15a7b000..0x15a7b1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 65 on 54310" daemon prio=10 tid=0x1703b000 nid=0x3fd 
waiting on condition [0x15acb000..0x15acc150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 64 on 54310" daemon prio=10 tid=0x1703a000 nid=0x3fc 
waiting on condition [0x15b1c000..0x15b1ced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 63 on 54310" daemon prio=10 tid=0x17038c00 nid=0x3fb 
waiting on condition [0x15b6d000..0x15b6de50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 62 on 54310" daemon prio=10 tid=0x17037c00 nid=0x3fa 
waiting on condition [0x15bbe000..0x15bbefd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 61 on 54310" daemon prio=10 tid=0x17036800 nid=0x3f9 
waiting on condition [0x15c0f000..0x15c0ff50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 60 on 54310" daemon prio=10 tid=0x17035800 nid=0x3f8 
waiting on condition [0x15c60000..0x15c610d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 59 on 54310" daemon prio=10 tid=0x17034400 nid=0x3f7 
waiting on condition [0x15cb1000..0x15cb2050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 58 on 54310" daemon prio=10 tid=0x17033400 nid=0x3f6 
waiting on condition [0x15d03000..0x15d031d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 57 on 54310" daemon prio=10 tid=0x17032000 nid=0x3f5 
waiting on condition [0x15d53000..0x15d54150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 56 on 54310" daemon prio=10 tid=0x17031000 nid=0x3f4 
waiting on condition [0x15da4000..0x15da4ed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 55 on 54310" daemon prio=10 tid=0x1702fc00 nid=0x3f3 
waiting on condition [0x15df5000..0x15df5e50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 54 on 54310" daemon prio=10 tid=0x1702ec00 nid=0x3f2 
waiting on condition [0x15e46000..0x15e46fd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 53 on 54310" daemon prio=10 tid=0x1702d800 nid=0x3f1 
waiting on condition [0x15e97000..0x15e97f50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 52 on 54310" daemon prio=10 tid=0x1702c800 nid=0x3f0 
waiting on condition [0x15ee8000..0x15ee90d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 51 on 54310" daemon prio=10 tid=0x1702b800 nid=0x3ef 
waiting on condition [0x15f39000..0x15f3a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 50 on 54310" daemon prio=10 tid=0x1702a400 nid=0x3ee 
waiting on condition [0x15f8b000..0x15f8b1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 49 on 54310" daemon prio=10 tid=0x17029400 nid=0x3ed 
waiting on condition [0x15fdb000..0x15fdc150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 48 on 54310" daemon prio=10 tid=0x17028000 nid=0x3ec 
waiting on condition [0x1602c000..0x1602ced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 47 on 54310" daemon prio=10 tid=0x17027000 nid=0x3eb 
waiting on condition [0x1607d000..0x1607de50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 46 on 54310" daemon prio=10 tid=0x173d1400 nid=0x3ea 
waiting on condition [0x160ce000..0x160cefd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 45 on 54310" daemon prio=10 tid=0x173d0000 nid=0x3e9 
waiting on condition [0x1611f000..0x1611ff50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 44 on 54310" daemon prio=10 tid=0x173cf000 nid=0x3e8 
waiting on condition [0x16170000..0x161710d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 43 on 54310" daemon prio=10 tid=0x173cdc00 nid=0x3e7 
waiting on condition [0x161c1000..0x161c2050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 42 on 54310" daemon prio=10 tid=0x173ccc00 nid=0x3e6 
waiting on condition [0x16213000..0x162131d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 41 on 54310" daemon prio=10 tid=0x173cb800 nid=0x3e5 
waiting on condition [0x16263000..0x16264150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 40 on 54310" daemon prio=10 tid=0x173ca800 nid=0x3e4 
waiting on condition [0x162b4000..0x162b4ed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 39 on 54310" daemon prio=10 tid=0x17373800 nid=0x3e3 
waiting on condition [0x16305000..0x16305e50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 38 on 54310" daemon prio=10 tid=0x17372800 nid=0x3e2 
waiting on condition [0x16356000..0x16356fd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 37 on 54310" daemon prio=10 tid=0x17371800 nid=0x3e1 
waiting on condition [0x163a7000..0x163a7f50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 36 on 54310" daemon prio=10 tid=0x17025400 nid=0x3e0 
waiting on condition [0x163f8000..0x163f90d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 35 on 54310" daemon prio=10 tid=0x17024000 nid=0x3df 
waiting on condition [0x16449000..0x1644a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 34 on 54310" daemon prio=10 tid=0x17022c00 nid=0x3de 
waiting on condition [0x1649b000..0x1649b1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 33 on 54310" daemon prio=10 tid=0x17021c00 nid=0x3dd 
waiting on condition [0x164eb000..0x164ec150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 32 on 54310" daemon prio=10 tid=0x17020800 nid=0x3dc 
waiting on condition [0x1653c000..0x1653ced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 31 on 54310" daemon prio=10 tid=0x1701f800 nid=0x3db 
waiting on condition [0x1658d000..0x1658de50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 30 on 54310" daemon prio=10 tid=0x1701e800 nid=0x3da 
waiting on condition [0x165de000..0x165defd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 29 on 54310" daemon prio=10 tid=0x1701d400 nid=0x3d9 
waiting on condition [0x1662f000..0x1662ff50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 28 on 54310" daemon prio=10 tid=0x1701c400 nid=0x3d8 
waiting on condition [0x16680000..0x166810d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 27 on 54310" daemon prio=10 tid=0x1701b000 nid=0x3d7 
waiting on condition [0x166d1000..0x166d2050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 26 on 54310" daemon prio=10 tid=0x1701a000 nid=0x3d6 
waiting on condition [0x16723000..0x167231d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 25 on 54310" daemon prio=10 tid=0x17018c00 nid=0x3d5 
waiting on condition [0x16773000..0x16774150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 24 on 54310" daemon prio=10 tid=0x17017c00 nid=0x3d4 
waiting on condition [0x167c4000..0x167c4ed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 23 on 54310" daemon prio=10 tid=0x17016800 nid=0x3d3 
waiting on condition [0x16815000..0x16815e50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 22 on 54310" daemon prio=10 tid=0x17015800 nid=0x3d2 
waiting on condition [0x16866000..0x16866fd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 21 on 54310" daemon prio=10 tid=0x17014400 nid=0x3d1 
waiting on condition [0x168b7000..0x168b7f50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 20 on 54310" daemon prio=10 tid=0x17013400 nid=0x3d0 
waiting on condition [0x16908000..0x169090d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 19 on 54310" daemon prio=10 tid=0x17012000 nid=0x3cf 
waiting on condition [0x16959000..0x1695a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 18 on 54310" daemon prio=10 tid=0x17011000 nid=0x3ce 
waiting on condition [0x169ab000..0x169ab1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 17 on 54310" daemon prio=10 tid=0x1700fc00 nid=0x3cd 
waiting on condition [0x169fb000..0x169fc150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 16 on 54310" daemon prio=10 tid=0x1700ec00 nid=0x3cc 
waiting on condition [0x16a4c000..0x16a4ced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 15 on 54310" daemon prio=10 tid=0x1700d800 nid=0x3cb 
waiting on condition [0x16a9d000..0x16a9de50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 14 on 54310" daemon prio=10 tid=0x1700c800 nid=0x3ca 
waiting on condition [0x16aee000..0x16aeefd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 13 on 54310" daemon prio=10 tid=0x1700b400 nid=0x3c9 
waiting on condition [0x16b3f000..0x16b3ff50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 12 on 54310" daemon prio=10 tid=0x1700a400 nid=0x3c8 
waiting on condition [0x16b90000..0x16b910d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 11 on 54310" daemon prio=10 tid=0x17009000 nid=0x3c7 
waiting on condition [0x16be1000..0x16be2050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 10 on 54310" daemon prio=10 tid=0x17008400 nid=0x3c6 
waiting on condition [0x16c33000..0x16c331d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 9 on 54310" daemon prio=10 tid=0x173d7000 nid=0x3c5 
waiting on condition [0x16c83000..0x16c84150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 8 on 54310" daemon prio=10 tid=0x173d5c00 nid=0x3c4 
waiting on condition [0x16cd4000..0x16cd4ed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 7 on 54310" daemon prio=10 tid=0x173fac00 nid=0x3c3 
waiting on condition [0x16d25000..0x16d25e50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 6 on 54310" daemon prio=10 tid=0x173f9c00 nid=0x3c2 
waiting on condition [0x16d76000..0x16d76fd0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 5 on 54310" daemon prio=10 tid=0x173fcc00 nid=0x3c1 
waiting on condition [0x16dc7000..0x16dc7f50]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 4 on 54310" daemon prio=10 tid=0x17303800 nid=0x3c0 
waiting on condition [0x16e18000..0x16e190d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 3 on 54310" daemon prio=10 tid=0x17324000 nid=0x3bf 
waiting on condition [0x16e69000..0x16e6a050]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 2 on 54310" daemon prio=10 tid=0x1733d400 nid=0x3be 
waiting on condition [0x16ebb000..0x16ebb1d0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 1 on 54310" daemon prio=10 tid=0x173ae800 nid=0x3bd 
waiting on condition [0x16f0b000..0x16f0c150]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server handler 0 on 54310" daemon prio=10 tid=0x17311400 nid=0x3bc 
waiting on condition [0x16f5c000..0x16f5ced0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1f7e2708> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:874)

   Locked ownable synchronizers:
    - None

"IPC Server listener on 54310" daemon prio=10 tid=0x173b1400 nid=0x3bb 
waiting on condition [0x16fad000..0x16fade50]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.ipc.Server$Listener.run(Server.java:319)

   Locked ownable synchronizers:
    - None

"IPC Server Responder" daemon prio=10 tid=0x17305000 nid=0x3ba runnable 
[0x16ffe000..0x16ffefd0]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x1f7e8b20> (a sun.nio.ch.Util$1)
    - locked <0x1f7e8b10> (a java.util.Collections$UnmodifiableSet)
    - locked <0x1f7e8928> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at org.apache.hadoop.ipc.Server$Responder.run(Server.java:455)

   Locked ownable synchronizers:
    - None

"SocketListener0-1" prio=10 tid=0x17308c00 nid=0x3b9 in Object.wait() 
[0x1715c000..0x1715cf50]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x34d7b800> (a org.mortbay.util.ThreadPool$PoolThread)
    at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
    - locked <0x34d7b800> (a org.mortbay.util.ThreadPool$PoolThread)

   Locked ownable synchronizers:
    - None

"Acceptor ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=50070]" 
prio=10 tid=0x1738b400 nid=0x3b7 runnable [0x1755c000..0x1755d050]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x34b77380> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
    at org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)

   Locked ownable synchronizers:
    - None

"SessionScavenger" daemon prio=10 tid=0x17309c00 nid=0x3b6 waiting on 
condition [0x171ff000..0x171ff1d0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)

   Locked ownable synchronizers:
    - None

"org.apache.hadoop.dfs.FSNamesystem$DecommissionedMonitor@922804" daemon 
prio=10 tid=0x09396c00 nid=0x3b5 waiting on condition 
[0x175ae000..0x175ae150]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.hadoop.dfs.FSNamesystem$DecommissionedMonitor.run(FSNamesystem.java:3602)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor@7cd37a" daemon 
prio=10 tid=0x09395800 nid=0x3b4 waiting on condition 
[0x175fe000..0x175feed0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x1fbfd5f8> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at 
org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor.run(FSNamesystem.java:1840)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"org.apache.hadoop.dfs.FSNamesystem$ReplicationMonitor@1c4f0f8" daemon 
prio=10 tid=0x09395000 nid=0x3b3 waiting on condition 
[0x17769000..0x17769e50]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.hadoop.dfs.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2191)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"org.apache.hadoop.dfs.FSNamesystem$HeartbeatMonitor@129b0e1" daemon 
prio=10 tid=0x08f64800 nid=0x3b1 waiting on condition 
[0x1785c000..0x1785cf50]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.hadoop.dfs.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2173)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"org.apache.hadoop.dfs.PendingReplicationBlocks$PendingReplicationMonitor@13f136e" 
daemon prio=10 tid=0x08f64000 nid=0x3b0 waiting on condition 
[0x1780b000..0x1780c0d0]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at 
org.apache.hadoop.dfs.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:185)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"Timer thread for monitoring dfs" daemon prio=10 tid=0x09641400 
nid=0x309 in Object.wait() [0x178ad000..0x178ade50]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1f7a9fc8> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x1f7a9fc8> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
    - None

"Timer thread for monitoring jvm" daemon prio=10 tid=0x0961e400 
nid=0x308 in Object.wait() [0x178fe000..0x178fefd0]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1f7b6f30> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x1f7b6f30> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

   Locked ownable synchronizers:
    - None

"RMI TCP Accept-0" daemon prio=10 tid=0x09607c00 nid=0x2d5 runnable 
[0x17b1d000..0x17b1e0d0]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    - locked <0x1f78e4b0> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:453)
    at java.net.ServerSocket.accept(ServerSocket.java:421)
    at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
    at 
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
    at java.lang.Thread.run(Thread.java:619)

   Locked ownable synchronizers:
    - None

"Low Memory Detector" daemon prio=10 tid=0x08a4cc00 nid=0x2be runnable 
[0x00000000..0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"CompilerThread1" daemon prio=10 tid=0x08a4b400 nid=0x2bd waiting on 
condition [0x00000000..0x19ac9698]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"CompilerThread0" daemon prio=10 tid=0x08a48c00 nid=0x2bc waiting on 
condition [0x00000000..0x19b4a618]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Signal Dispatcher" daemon prio=10 tid=0x08a47800 nid=0x2bb runnable 
[0x00000000..0x19b9bbb0]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

"Finalizer" daemon prio=10 tid=0x08a2c800 nid=0x2ba in Object.wait() 
[0x19dec000..0x19dece50]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1f7effd0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x1f7effd0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

   Locked ownable synchronizers:
    - None

"Reference Handler" daemon prio=10 tid=0x08a2b400 nid=0x2b9 in 
Object.wait() [0x19e3d000..0x19e3dfd0]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1f7e6830> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x1f7e6830> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
    - None

"main" prio=10 tid=0x08941c00 nid=0x2af in Object.wait() 
[0xb7f14000..0xb7f142a8]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x1f7bc1a0> (a org.apache.hadoop.ipc.RPC$Server)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.hadoop.ipc.Server.join(Server.java:1021)
    - locked <0x1f7bc1a0> (a org.apache.hadoop.ipc.RPC$Server)
    at org.apache.hadoop.dfs.NameNode.join(NameNode.java:206)
    at org.apache.hadoop.dfs.NameNode.main(NameNode.java:841)

   Locked ownable synchronizers:
    - None

"VM Thread" prio=10 tid=0x08a28400 nid=0x2b8 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x08948400 nid=0x2b0 runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x08949800 nid=0x2b1 runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x0894a800 nid=0x2b2 runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0894b800 nid=0x2b3 runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0894c800 nid=0x2b4 runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0894dc00 nid=0x2b5 runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0894ec00 nid=0x2b6 runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0894fc00 nid=0x2b7 runnable

"VM Periodic Task Thread" prio=10 tid=0x09611000 nid=0x2d6 waiting on 
condition

JNI global references: 1178













Raghu Angadi wrote:
>
> Give an example stack trace of a thread that is blocked on callQ and a 
> handler thread that is not. trace of all the threads would be even 
> better.
>
> If your handlers waiting on callQ, then they are waiting for work.
>
> Raghu.
>
> Sagar Naik wrote:
>> *Problem:*
>> The "ls" is taking noticeable  time to respond.
>> *system:*
>> I have about 1.6 million files and namenode is prety much full with 
>> heap(2400MB).
>>
>> I have configured dfs.handler.count to 100, and all IPC server 
>> threads are locked onto a Queue (I think callQueue)
>>
>> In the namenode logs, I occasionally see Out of memory Exception.
>>
>> Any suggestions, to improve the response time of system ?
>> I can increase the dfs.handler.count to 256, but I guess I will see 
>> more OOM exceptions.
>>
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message