hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: NameNode deadlocked (help?)
Date Sat, 15 May 2010 01:27:55 GMT
Hey Brian,

Yep, excessive GC definitely sounds like a likely culprit. I'm surprised you
didn't see OOMEs in the log, though.

If you want to monitor GC, I'd recommend adding -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps to your java options -
occasionally useful for times like this.

Glad to hear you're back up and running.

-Todd

On Fri, May 14, 2010 at 5:53 PM, Brian Bockelman <bbockelm@cse.unl.edu>wrote:

> Hey Todd,
>
> The thread remains in that trace after a few attempts.
>
> Being that no one was going to get anything done quickly, we decided to
> restart the NN.  Upon restart, I found something *really* interesting - the
> server had been running for quite awhile with 1GB of heap!
>
> Being that these are the statistics for the cluster:
>
> 720246 files and directories, 3096099 blocks = 3816345 total
>
> I have no clue how this system ran before!  Hopefully this is really the
> issue and not some red herring.
>
> My theory is that we somehow managed to scrape below the memory limits ...
> the same sysadmin has been complaining for a few months that the NN would
> lock up for a few seconds at 100% CPU.  This could have been signs that I
> (stupidly) missed of excessive garbage collection.
>
> Sigh.  Need a beer.
>
> Brian
>
> On May 14, 2010, at 7:46 PM, Todd Lipcon wrote:
>
> > Hey Brian,
> >
> > I don't see a way in which this could infinite loop unless some
> > datastructure got corrupted.
> >
> > If you jstack the NN a few times, does that thread remain in that trace
> or
> > does it move around?
> >
> > -Todd
> >
> > On Fri, May 14, 2010 at 5:41 PM, Todd Lipcon <todd@cloudera.com> wrote:
> >
> >> Hey Brian,
> >>
> >> Looks like it's not deadlocked, but rather just busy doing a lot of
> work:
> >>
> >>
> >>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255
> "
> >> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> >> [0x00000000413da000..0x00000000413daa10]
> >>  java.lang.Thread.State: RUNNABLE
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
> >>       - locked <0x00002aaab3843e40> (a
> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
> >>       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
> >>       - locked <0x00002aaab3653848> (a java.util.ArrayList)
> >>       - locked <0x00002aaab3843e40> (a
> >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>       at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
> >>       at java.lang.Thread.run(Thread.java:619)
> >>
> >> Let me dig up the 19.1 source code and see if this looks like an
> infinite
> >> loop or just one that's tying it up for some number of seconds.
> >>
> >> Anything being written into the logs?
> >>
> >> -Todd
> >>
> >>
> >> On Fri, May 14, 2010 at 5:22 PM, Brian Bockelman <bbockelm@cse.unl.edu
> >wrote:
> >>
> >>> Hey guys,
> >>>
> >>> I know it's 5PM on a Friday, but we just saw one of our big cluster's
> >>> namenode's deadlock.
> >>>
> >>> This is 0.19.1; does this ring a bell for anyone?  I haven't had any
> time
> >>> to start going through source code, but I figured I'd send out a SOS in
> case
> >>> if this looked familiar.
> >>>
> >>> We had restarted this cluster a few hours ago and made the following
> >>> changes:
> >>>
> >>> 1) Increased the number of datanode handlers from 10 to 40.
> >>> 2) Increased ipc.server.listen.queue.size from 128 to 256.
> >>>
> >>> If nothing else, I figure a deadlocked NN might be interesting to
> devs...
> >>>
> >>> Brian
> >>>
> >>> 2010-05-14 17:11:30
> >>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.2-b01 mixed
> mode):
> >>>
> >>> "IPC Server handler 39 on 9000" daemon prio=10 tid=0x00002aaafc181400
> >>> nid=0x4cd waiting for monitor entry
> [0x0000000045962000..0x0000000045962d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 38 on 9000" daemon prio=10 tid=0x00002aaafc17f800
> >>> nid=0x4cc waiting for monitor entry
> [0x0000000045861000..0x0000000045861d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getStats(NameNode.java:505)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 37 on 9000" daemon prio=10 tid=0x00002aaafc17e000
> >>> nid=0x4cb waiting for monitor entry
> [0x000000004575f000..0x0000000045760a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 36 on 9000" daemon prio=10 tid=0x00002aaafc17c400
> >>> nid=0x4ca waiting for monitor entry
> [0x000000004565e000..0x000000004565fa10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 35 on 9000" daemon prio=10 tid=0x00002aaafc17a800
> >>> nid=0x4c9 waiting for monitor entry
> [0x000000004555e000..0x000000004555eb90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 34 on 9000" daemon prio=10 tid=0x00002aaafc178c00
> >>> nid=0x4c8 waiting for monitor entry
> [0x000000004545d000..0x000000004545db10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 33 on 9000" daemon prio=10 tid=0x00002aaafc177400
> >>> nid=0x4c7 waiting for monitor entry
> [0x000000004535c000..0x000000004535cc90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 32 on 9000" daemon prio=10 tid=0x00002aaafc175800
> >>> nid=0x4c6 waiting for monitor entry
> [0x000000004525a000..0x000000004525bc10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 31 on 9000" daemon prio=10 tid=0x00002aaafc173c00
> >>> nid=0x4c5 waiting for monitor entry
> [0x000000004515a000..0x000000004515ad90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:3281)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:649)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 30 on 9000" daemon prio=10 tid=0x00002aaafc172400
> >>> nid=0x4c4 waiting for monitor entry
> [0x0000000045059000..0x0000000045059d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 29 on 9000" daemon prio=10 tid=0x00002aaafc171000
> >>> nid=0x4c3 waiting for monitor entry
> [0x0000000044f57000..0x0000000044f58a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 28 on 9000" daemon prio=10 tid=0x00002aaafc16f400
> >>> nid=0x4c2 waiting for monitor entry
> [0x0000000044e56000..0x0000000044e57a10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 27 on 9000" daemon prio=10 tid=0x00002aaafc16d800
> >>> nid=0x4c1 waiting for monitor entry
> [0x0000000044d56000..0x0000000044d56b90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 26 on 9000" daemon prio=10 tid=0x00002aaafc16bc00
> >>> nid=0x4c0 waiting for monitor entry
> [0x0000000044c55000..0x0000000044c55b10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 25 on 9000" daemon prio=10 tid=0x00002aaafc16a400
> >>> nid=0x4bf waiting for monitor entry
> [0x0000000044b54000..0x0000000044b54c90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 24 on 9000" daemon prio=10 tid=0x00002aaafc168800
> >>> nid=0x4be waiting for monitor entry
> [0x0000000044a53000..0x0000000044a53c10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 23 on 9000" daemon prio=10 tid=0x00002aaafc166c00
> >>> nid=0x4bd waiting for monitor entry
> [0x0000000044952000..0x0000000044952d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 22 on 9000" daemon prio=10 tid=0x00002aaafc165400
> >>> nid=0x4bc waiting for monitor entry
> [0x0000000044851000..0x0000000044851d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 21 on 9000" daemon prio=10 tid=0x00002aaafc164000
> >>> nid=0x4bb waiting for monitor entry
> [0x0000000044750000..0x0000000044750a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 20 on 9000" daemon prio=10 tid=0x00002aaafc162400
> >>> nid=0x4ba waiting for monitor entry
> [0x000000004464f000..0x000000004464fa10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 19 on 9000" daemon prio=10 tid=0x00002aaafc488400
> >>> nid=0x4b9 waiting for monitor entry
> [0x000000004454e000..0x000000004454eb90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 18 on 9000" daemon prio=10 tid=0x00002aaafc486800
> >>> nid=0x4b8 waiting for monitor entry
> [0x000000004444d000..0x000000004444db10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 17 on 9000" daemon prio=10 tid=0x00002aaafc484c00
> >>> nid=0x4b7 waiting for monitor entry
> [0x000000004434c000..0x000000004434cc90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBlock(FSNamesystem.java:4548)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode.java:402)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 16 on 9000" daemon prio=10 tid=0x00002aaafc483000
> >>> nid=0x4b6 waiting for monitor entry
> [0x000000004424a000..0x000000004424bc10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 15 on 9000" daemon prio=10 tid=0x00002aaafc481800
> >>> nid=0x4b5 waiting for monitor entry
> [0x000000004414a000..0x000000004414ad90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 14 on 9000" daemon prio=10 tid=0x00002aaafc47fc00
> >>> nid=0x4b4 waiting for monitor entry
> [0x0000000044049000..0x0000000044049d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 13 on 9000" daemon prio=10 tid=0x00002aaafc47e000
> >>> nid=0x4b3 waiting for monitor entry
> [0x0000000043f48000..0x0000000043f48a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.handleHeartbeat(FSNamesystem.java:2231)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:625)
> >>>       at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 12 on 9000" daemon prio=10 tid=0x00002aaafc47c400
> >>> nid=0x4b2 waiting for monitor entry
> [0x0000000043e46000..0x0000000043e47a10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 11 on 9000" daemon prio=10 tid=0x00002aaafc47b400
> >>> nid=0x4b1 waiting for monitor entry
> [0x0000000043d45000..0x0000000043d46b90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 10 on 9000" daemon prio=10 tid=0x00002aaafc479800
> >>> nid=0x4b0 waiting for monitor entry
> [0x0000000043c45000..0x0000000043c45b10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1294)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:351)
> >>>       at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 9 on 9000" daemon prio=10 tid=0x00002aaafc477c00
> >>> nid=0x4af waiting for monitor entry
> [0x0000000043b43000..0x0000000043b44c90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 8 on 9000" daemon prio=10 tid=0x00002aaafc476400
> >>> nid=0x4ae waiting for monitor entry
> [0x0000000043a42000..0x0000000043a43c10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 7 on 9000" daemon prio=10 tid=0x00002aaafc474c00
> >>> nid=0x4ad waiting for monitor entry
> [0x0000000043942000..0x0000000043942d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 6 on 9000" daemon prio=10 tid=0x00002aaafc473000
> >>> nid=0x4ac waiting for monitor entry
> [0x0000000043841000..0x0000000043841d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 5 on 9000" daemon prio=10 tid=0x00002aaafc471400
> >>> nid=0x4ab waiting for monitor entry
> [0x0000000043740000..0x0000000043740a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 4 on 9000" daemon prio=10 tid=0x00002aaafc121800
> >>> nid=0x4aa waiting for monitor entry
> [0x000000004363f000..0x000000004363fa10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 3 on 9000" daemon prio=10 tid=0x00002aaafc11fc00
> >>> nid=0x4a9 waiting for monitor entry
> [0x000000004353d000..0x000000004353eb90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 2 on 9000" daemon prio=10 tid=0x00002aaafc11e400
> >>> nid=0x4a8 waiting for monitor entry
> [0x000000004343c000..0x000000004343db10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 1 on 9000" daemon prio=10 tid=0x00002aaafc467800
> >>> nid=0x4a7 waiting for monitor entry
> [0x000000004333b000..0x000000004333cc90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>       at
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server handler 0 on 9000" daemon prio=10 tid=0x00002aaafc466400
> >>> nid=0x4a6 waiting for monitor entry
> [0x000000004323b000..0x000000004323bc10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInternal(FSNamesystem.java:801)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:784)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:751)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.NameNode.getBlockLocations(NameNode.java:272)
> >>>       at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> >>>       at
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>       at java.lang.reflect.Method.invoke(Method.java:597)
> >>>       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> >>>       at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> >>>
> >>> "IPC Server listener on 9000" daemon prio=10 tid=0x00002aaafc12fc00
> >>> nid=0x4a5 waiting on condition [0x000000004313a000..0x000000004313ad90]
> >>>  java.lang.Thread.State: WAITING (parking)
> >>>       at sun.misc.Unsafe.park(Native Method)
> >>>       - parking to wait for  <0x00002aaab3b8fd70> (a
> >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> >>>       at
> >>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >>>       at
> >>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
> >>>       at
> >>>
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:254)
> >>>       at
> >>> org.apache.hadoop.ipc.Server$Connection.processData(Server.java:850)
> >>>       at
> >>> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:814)
> >>>       at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:400)
> >>>       at org.apache.hadoop.ipc.Server$Listener.run(Server.java:309)
> >>>
> >>> "IPC Server Responder" daemon prio=10 tid=0x00002aaafc0bbc00 nid=0x4a4
> >>> runnable [0x0000000043039000..0x0000000043039d10]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>       at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >>>       at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> >>>       at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> >>>       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> >>>       - locked <0x00002aaab3b90e80> (a sun.nio.ch.Util$1)
> >>>       - locked <0x00002aaab3b90e68> (a
> >>> java.util.Collections$UnmodifiableSet)
> >>>       - locked <0x00002aaab3b90b08> (a sun.nio.ch.EPollSelectorImpl)
> >>>       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> >>>       at org.apache.hadoop.ipc.Server$Responder.run(Server.java:458)
> >>>
> >>> "SocketListener0-1" prio=10 tid=0x00002aaafc0ba400 nid=0x4a3 in
> >>> Object.wait() [0x0000000042f38000..0x0000000042f38a90]
> >>>  java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaadf36cb00> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>       at
> org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
> >>>       - locked <0x00002aaadf36cb00> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>
> >>> "SocketListener0-0" prio=10 tid=0x00002aaafc0b4c00 nid=0x4a2 in
> >>> Object.wait() [0x0000000042e37000..0x0000000042e37a10]
> >>>  java.lang.Thread.State: TIMED_WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaadf36c820> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>       at
> org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:522)
> >>>       - locked <0x00002aaadf36c820> (a
> >>> org.mortbay.util.ThreadPool$PoolThread)
> >>>
> >>> "Acceptor ServerSocket[addr=proxy-1/169.228.130.63
> ,port=0,localport=50070]"
> >>> prio=10 tid=0x00002aaafc0c8400 nid=0x4a1 runnable
> >>> [0x0000000042d36000..0x0000000042d36b90]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>       at java.lang.Object.<init>(Object.java:20)
> >>>       at java.net.SocketImpl.<init>(SocketImpl.java:27)
> >>>       at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
> >>>       at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
> >>>       at java.net.Socket.setImpl(Socket.java:434)
> >>>       at java.net.ServerSocket.implAccept(ServerSocket.java:445)
> >>>       at java.net.ServerSocket.accept(ServerSocket.java:421)
> >>>       at
> >>> org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
> >>>       at
> >>> org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
> >>>
> >>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0c1000 nid=0x4a0
> waiting
> >>> on condition [0x0000000042c35000..0x0000000042c35b10]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
> >>>
> >>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc0fa000 nid=0x49f
> waiting
> >>> on condition [0x0000000040b2a000..0x0000000040b2ac90]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
> >>>
> >>> "SessionScavenger" daemon prio=10 tid=0x00002aaafc139800 nid=0x49a
> waiting
> >>> on condition [0x00000000408b7000..0x00000000408b7c10]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:587)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1b0deb5f
> "
> >>> daemon prio=10 tid=0x00002aaafc0a2800 nid=0x496 waiting for monitor
> entry
> >>> [0x0000000042b34000..0x0000000042b34d90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:61)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@784a7df6
> "
> >>> daemon prio=10 tid=0x00002aaafc090c00 nid=0x495 waiting for monitor
> entry
> >>> [0x00000000411f8000..0x00000000411f8d10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeReplicationWork(FSNamesystem.java:2410)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.computeDatanodeWork(FSNamesystem.java:2376)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2333)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>> "org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@d24111a"
> >>> daemon prio=10 tid=0x00002aaafc013400 nid=0x494 waiting for monitor
> entry
> >>> [0x00000000410f7000..0x00000000410f7a90]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:349)
> >>>       - waiting to lock <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@1c778255
> "
> >>> daemon prio=10 tid=0x00002aaafc012c00 nid=0x493 runnable
> >>> [0x00000000413da000..0x00000000413daa10]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeStoredBlock(FSNamesystem.java:3236)
> >>>       - locked <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeDatanode(FSNamesystem.java:2695)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.heartbeatCheck(FSNamesystem.java:2785)
> >>>       - locked <0x00002aaab3659dd8> (a java.util.TreeMap)
> >>>       - locked <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       - locked <0x00002aaab3843e40> (a
> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2312)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>>
> >>>
> "org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@44676e3f
> "
> >>> daemon prio=10 tid=0x00002aaafc012800 nid=0x492 waiting on condition
> >>> [0x0000000040d16000..0x0000000040d16b90]
> >>>  java.lang.Thread.State: TIMED_WAITING (sleeping)
> >>>       at java.lang.Thread.sleep(Native Method)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> >>>       at java.lang.Thread.run(Thread.java:619)
> >>>
> >>> "Timer thread for monitoring dfs" daemon prio=10 tid=0x00002aaafc03d800
> >>> nid=0x428 waiting for monitor entry
> [0x0000000040e7e000..0x0000000040e7ed10]
> >>>  java.lang.Thread.State: BLOCKED (on object monitor)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStats(FSNamesystem.java:3326)
> >>>       - waiting to lock <0x00002aaab3653848> (a java.util.ArrayList)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getCapacityTotal(FSNamesystem.java:3335)
> >>>       at
> >>>
> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics.doUpdates(FSNamesystemMetrics.java:90)
> >>>       - locked <0x00002aaab3b3c580> (a
> >>> org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics)
> >>>       at
> >>>
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.timerEvent(AbstractMetricsContext.java:284)
> >>>       at
> >>>
> org.apache.hadoop.metrics.spi.AbstractMetricsContext.access$000(AbstractMetricsContext.java:50)
> >>>       at
> >>>
> org.apache.hadoop.metrics.spi.AbstractMetricsContext$1.run(AbstractMetricsContext.java:249)
> >>>       at java.util.TimerThread.mainLoop(Timer.java:512)
> >>>       at java.util.TimerThread.run(Timer.java:462)
> >>>
> >>> "Low Memory Detector" daemon prio=10 tid=0x000000005c306c00 nid=0x426
> >>> runnable [0x0000000000000000..0x0000000000000000]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "CompilerThread1" daemon prio=10 tid=0x000000005c304000 nid=0x425
> waiting
> >>> on condition [0x0000000000000000..0x0000000042830420]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "CompilerThread0" daemon prio=10 tid=0x000000005c300000 nid=0x424
> waiting
> >>> on condition [0x0000000000000000..0x000000004272f3a0]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "Signal Dispatcher" daemon prio=10 tid=0x000000005c2fe400 nid=0x423
> >>> waiting on condition [0x0000000000000000..0x0000000000000000]
> >>>  java.lang.Thread.State: RUNNABLE
> >>>
> >>> "Finalizer" daemon prio=10 tid=0x000000005c2da800 nid=0x422 in
> >>> Object.wait() [0x000000004252e000..0x000000004252ec10]
> >>>  java.lang.Thread.State: WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaab38ec320> (a
> >>> java.lang.ref.ReferenceQueue$Lock)
> >>>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> >>>       - locked <0x00002aaab38ec320> (a
> java.lang.ref.ReferenceQueue$Lock)
> >>>       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> >>>       at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> >>>
> >>> "Reference Handler" daemon prio=10 tid=0x000000005c2d8c00 nid=0x421 in
> >>> Object.wait() [0x000000004177e000..0x000000004177ed90]
> >>>  java.lang.Thread.State: WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaab38e9ec0> (a
> java.lang.ref.Reference$Lock)
> >>>       at java.lang.Object.wait(Object.java:485)
> >>>       at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> >>>       - locked <0x00002aaab38e9ec0> (a java.lang.ref.Reference$Lock)
> >>>
> >>> "main" prio=10 tid=0x000000005c268400 nid=0x417 in Object.wait()
> >>> [0x0000000040494000..0x0000000040494ed0]
> >>>  java.lang.Thread.State: WAITING (on object monitor)
> >>>       at java.lang.Object.wait(Native Method)
> >>>       - waiting on <0x00002aaab3b3db50> (a
> >>> org.apache.hadoop.ipc.RPC$Server)
> >>>       at java.lang.Object.wait(Object.java:485)
> >>>       at org.apache.hadoop.ipc.Server.join(Server.java:1023)
> >>>       - locked <0x00002aaab3b3db50> (a
> org.apache.hadoop.ipc.RPC$Server)
> >>>       at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode.join(NameNode.java:221)
> >>>       at
> >>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:870)
> >>>
> >>> "VM Thread" prio=10 tid=0x000000005c2d3400 nid=0x420 runnable
> >>>
> >>> "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005c273000
> nid=0x418
> >>> runnable
> >>>
> >>> "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005c274800
> nid=0x419
> >>> runnable
> >>>
> >>> "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005c276400
> nid=0x41a
> >>> runnable
> >>>
> >>> "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005c277c00
> nid=0x41b
> >>> runnable
> >>>
> >>> "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000005c279400
> nid=0x41c
> >>> runnable
> >>>
> >>> "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000005c27b000
> nid=0x41d
> >>> runnable
> >>>
> >>> "GC task thread#6 (ParallelGC)" prio=10 tid=0x000000005c27c800
> nid=0x41e
> >>> runnable
> >>>
> >>> "GC task thread#7 (ParallelGC)" prio=10 tid=0x000000005c27e000
> nid=0x41f
> >>> runnable
> >>>
> >>> "VM Periodic Task Thread" prio=10 tid=0x000000005c308c00 nid=0x427
> waiting
> >>> on condition
> >>>
> >>> JNI global references: 984
> >>>
> >>> Heap
> >>> PSYoungGen      total 203712K, used 107631K [0x00002aaadd0b0000,
> >>> 0x00002aaaf1e00000, 0x00002aaaf1e00000)
> >>> eden space 124096K, 86% used
> >>> [0x00002aaadd0b0000,0x00002aaae39cbe58,0x00002aaae49e0000)
> >>> from space 79616K, 0% used
> >>> [0x00002aaae49e0000,0x00002aaae49e0000,0x00002aaae97a0000)
> >>> to   space 113728K, 0% used
> >>> [0x00002aaaeaef0000,0x00002aaaeaef0000,0x00002aaaf1e00000)
> >>> PSOldGen        total 682688K, used 682687K [0x00002aaab3600000,
> >>> 0x00002aaadd0b0000, 0x00002aaadd0b0000)
> >>> object space 682688K, 99% used
> >>> [0x00002aaab3600000,0x00002aaadd0affe8,0x00002aaadd0b0000)
> >>> PSPermGen       total 21248K, used 16689K [0x00002aaaae200000,
> >>> 0x00002aaaaf6c0000, 0x00002aaab3600000)
> >>> object space 21248K, 78% used
> >>> [0x00002aaaae200000,0x00002aaaaf24c660,0x00002aaaaf6c0000)
> >>>
> >>>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message