hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: NameNode deadlocked (help?)
Date Sat, 15 May 2010 00:46:04 GMT
Hey Brian,

I don't see a way in which this could infinite loop unless some
datastructure got corrupted.

If you jstack the NN a few times, does that thread remain in that trace or
does it move around?

-Todd

On Fri, May 14, 2010 at 5:41 PM, Todd Lipcon <todd@cloudera.com> wrote:

> Hey Brian,
>
> Looks like it's not deadlocked, but rather just busy doing a lot of work:
>
>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> [0x00000000413da000..0x00000000413daa10]
>   java.lang.Thread.State: RUNNABLE
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>        - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>        - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>        - locked <0x00002aaab3653848> (a java.util.ArrayList)
>        - locked <0x00002aaab3843e40> (a
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>        at java.lang.Thread.run(Thread.java:619)
>
> Let me dig up the 19.1 source code and see if this looks like an infinite
> loop or just one that's tying it up for some number of seconds.
>
> Anything being written into the logs?
>
> -Todd
>
>
> On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bbockelm@cse.unl.edu>wrote:
>
>> Hey guys,
>>
>> I know it's 5PM on a Friday, but we just saw one of our big cluster's
>> namenode's deadlock.
>>
>> This is 0.19.1; does this ring a bell for anyone?  I haven't had any time
>> to start going through source code, but I figured I'd send out a SOS in case
>> if this looked familiar.
>>
>> We had restarted this cluster a few hours ago and made the following
>> changes:
>>
>> 1) Increased the number of datanode handlers from 10 to 40.
>> 2) Increased ipc.server.listen.queue.size from 128 to 256.
>>
>> If nothing else, I figure a deadlocked NN might be interesting to devs...
>>
>> Brian
>>
>> 2010-05-14 17:11:30
>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed mode):
>>
>> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
>> nid=0x4cd waiting for monitor entry [0x0000000045962000..0x0000000045962d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
>> nid=0x4cc waiting for monitor entry [0x0000000045861000..0x0000000045861d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
>> nid=0x4cb waiting for monitor entry [0x000000004575f000..0x0000000045760a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
>> nid=0x4ca waiting for monitor entry [0x000000004565e000..0x000000004565fa10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
>> nid=0x4c9 waiting for monitor entry [0x000000004555e000..0x000000004555eb90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
>> nid=0x4c8 waiting for monitor entry [0x000000004545d000..0x000000004545db10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
>> nid=0x4c7 waiting for monitor entry [0x000000004535c000..0x000000004535cc90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
>> nid=0x4c6 waiting for monitor entry [0x000000004525a000..0x000000004525bc10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
>> nid=0x4c5 waiting for monitor entry [0x000000004515a000..0x000000004515ad90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
>> nid=0x4c4 waiting for monitor entry [0x0000000045059000..0x0000000045059d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
>> nid=0x4c3 waiting for monitor entry [0x0000000044f57000..0x0000000044f58a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
>> nid=0x4c2 waiting for monitor entry [0x0000000044e56000..0x0000000044e57a10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
>> nid=0x4c1 waiting for monitor entry [0x0000000044d56000..0x0000000044d56b90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
>> nid=0x4c0 waiting for monitor entry [0x0000000044c55000..0x0000000044c55b10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
>> nid=0x4bf waiting for monitor entry [0x0000000044b54000..0x0000000044b54c90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
>> nid=0x4be waiting for monitor entry [0x0000000044a53000..0x0000000044a53c10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
>> nid=0x4bd waiting for monitor entry [0x0000000044952000..0x0000000044952d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
>> nid=0x4bc waiting for monitor entry [0x0000000044851000..0x0000000044851d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
>> nid=0x4bb waiting for monitor entry [0x0000000044750000..0x0000000044750a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
>> nid=0x4ba waiting for monitor entry [0x000000004464f000..0x000000004464fa10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
>> nid=0x4b9 waiting for monitor entry [0x000000004454e000..0x000000004454eb90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
>> nid=0x4b8 waiting for monitor entry [0x000000004444d000..0x000000004444db10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
>> nid=0x4b7 waiting for monitor entry [0x000000004434c000..0x000000004434cc90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
>> nid=0x4b6 waiting for monitor entry [0x000000004424a000..0x000000004424bc10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
>> nid=0x4b5 waiting for monitor entry [0x000000004414a000..0x000000004414ad90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
>> nid=0x4b4 waiting for monitor entry [0x0000000044049000..0x0000000044049d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
>> nid=0x4b3 waiting for monitor entry [0x0000000043f48000..0x0000000043f48a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
>>        at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
>> nid=0x4b2 waiting for monitor entry [0x0000000043e46000..0x0000000043e47a10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
>> nid=0x4b1 waiting for monitor entry [0x0000000043d45000..0x0000000043d46b90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
>> nid=0x4b0 waiting for monitor entry [0x0000000043c45000..0x0000000043c45b10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
>>        at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
>> nid=0x4af waiting for monitor entry [0x0000000043b43000..0x0000000043b44c90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
>> nid=0x4ae waiting for monitor entry [0x0000000043a42000..0x0000000043a43c10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
>> nid=0x4ad waiting for monitor entry [0x0000000043942000..0x0000000043942d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
>> nid=0x4ac waiting for monitor entry [0x0000000043841000..0x0000000043841d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
>> nid=0x4ab waiting for monitor entry [0x0000000043740000..0x0000000043740a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
>> nid=0x4aa waiting for monitor entry [0x000000004363f000..0x000000004363fa10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
>> nid=0x4a9 waiting for monitor entry [0x000000004353d000..0x000000004353eb90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
>> nid=0x4a8 waiting for monitor entry [0x000000004343c000..0x000000004343db10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
>> nid=0x4a7 waiting for monitor entry [0x000000004333b000..0x000000004333cc90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
>> nid=0x4a6 waiting for monitor entry [0x000000004323b000..0x000000004323bc10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
>>        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>>
>> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
>> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
>>   java.lang.Thread.State: WAITING (parking)
>>        at sun.misc.Unsafe.park(Native Method)
>>        - parking to wait for  <0x00002aaab3b8fd70> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>        at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>        at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>>        at
>> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
>>        at
>> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
>>        at
>> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
>>        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
>>        at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
>>
>> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
>> runnable [0x0000000043039000..0x0000000043039d10]
>>   java.lang.Thread.State: RUNNABLE
>>        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>>        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>        - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
>>        - locked <0x00002aaab3b90e68> (a
>> java.util.Collections$UnmodifiableSet)
>>        - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
>>        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>        at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
>>
>> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
>> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaadf36cb00> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>        - locked <0x00002aaadf36cb00> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>
>> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
>> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
>>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaadf36c820> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
>>        - locked <0x00002aaadf36c820> (a
>> org.mortbay.util.ThreadPool$PoolThread)
>>
>> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63,port=0,localport=50070]"
>> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
>> [0x0000000042d36000..0x0000000042d36b90]
>>   java.lang.Thread.State: RUNNABLE
>>        at java.lang.Object.<init>(Object.java:20)
>>        at java.net.SocketImpl.<init>(SocketImpl.java:27)
>>        at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>>        at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>>        at java.net.Socket.setImpl(Socket.java:434)
>>        at java.net.ServerSocket.implAccept(ServerSocket.java:445)
>>        at java.net.ServerSocket.accept(ServerSocket.java:421)
>>        at
>> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
>>        at
>> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
>>
>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0 waiting
>> on condition [0x0000000042c35000..0x0000000042c35b10]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>
>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f waiting
>> on condition [0x0000000040b2a000..0x0000000040b2ac90]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>
>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a waiting
>> on condition [0x00000000408b7000..0x00000000408b7c10]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f"
>> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor entry
>> [0x0000000042b34000..0x0000000042b34d90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6"
>> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor entry
>> [0x00000000411f8000..0x00000000411f8d10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
>> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor entry
>> [0x00000000410f7000..0x00000000410f7a90]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
>>        - waiting to lock <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255"
>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
>> [0x00000000413da000..0x00000000413daa10]
>>   java.lang.Thread.State: RUNNABLE
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
>>        - locked <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
>>        - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
>>        - locked <0x00002aaab3653848> (a java.util.ArrayList)
>>        - locked <0x00002aaab3843e40> (a
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>>
>> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f"
>> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
>> [0x0000000040d16000..0x0000000040d16b90]
>>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>>        at java.lang.Thread.sleep(Native Method)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>>        at java.lang.Thread.run(Thread.java:619)
>>
>> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
>> nid=0x428 waiting for monitor entry [0x0000000040e7e000..0x0000000040e7ed10]
>>   java.lang.Thread.State: BLOCKED (on object monitor)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
>>        - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
>>        - locked <0x00002aaab3b3c580> (a
>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
>>        at
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
>>        at
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
>>        at
>> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
>>        at java.util.TimerThread.mainLoop(Timer.java:512)
>>        at java.util.TimerThread.run(Timer.java:462)
>>
>> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
>> runnable [0x0000000000000000..0x0000000000000000]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425 waiting
>> on condition [0x0000000000000000..0x0000000042830420]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424 waiting
>> on condition [0x0000000000000000..0x000000004272f3a0]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423
>> waiting on condition [0x0000000000000000..0x0000000000000000]
>>   java.lang.Thread.State: RUNNABLE
>>
>> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
>> Object.wait() [0x000000004252e000..0x000000004252ec10]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaab38ec320> (a
>> java.lang.ref.ReferenceQueue$Lock)
>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
>>        - locked <0x00002aaab38ec320> (a java.lang.ref.ReferenceQueue$Lock)
>>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
>>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>>
>> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
>> Object.wait() [0x000000004177e000..0x000000004177ed90]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>        at java.lang.Object.wait(Object.java:485)
>>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>>        - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
>>
>> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
>> [0x0000000040494000..0x0000000040494ed0]
>>   java.lang.Thread.State: WAITING (on object monitor)
>>        at java.lang.Object.wait(Native Method)
>>        - waiting on <0x00002aaab3b3db50> (a
>> org.apache.hadoop.ipc.RPC$Server)
>>        at java.lang.Object.wait(Object.java:485)
>>        at org.apache.hadoop.ipc.Server.join(Server.java:1023)
>>        - locked <0x00002aaab3b3db50> (a org.apache.hadoop.ipc.RPC$Server)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
>>
>> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
>>
>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000 nid=0x418
>> runnable
>>
>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800 nid=0x419
>> runnable
>>
>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400 nid=0x41a
>> runnable
>>
>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00 nid=0x41b
>> runnable
>>
>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400 nid=0x41c
>> runnable
>>
>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000 nid=0x41d
>> runnable
>>
>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800 nid=0x41e
>> runnable
>>
>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000 nid=0x41f
>> runnable
>>
>> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427 waiting
>> on condition
>>
>> JNI global references: 984
>>
>> Heap
>>  PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
>> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
>>  eden space 124096K, 86% used
>> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
>>  from space 79616K, 0% used
>> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
>>  to   space 113728K, 0% used
>> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
>>  PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
>> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
>>  object space 682688K, 99% used
>> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
>>  PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
>> 0x00002aaaaf6c0000, 0x00002aaab3600000)
>>  object space 21248K, 78% used
>> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message