hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-870) locked up master
Date Thu, 04 Sep 2008 18:41:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628429#action_12628429
] 

stack commented on HBASE-870:
-----------------------------

I lost the thread dumps.  Here are snippets copied from my terminal:
{code}
Full thread dump Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode):

"SocketListener0-4" prio=10 tid=0x00007f9ad003a000 nid=0x5f9a waiting for monitor entry [0x00000000415b9000..0x00000000415baa00]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:474)
        - waiting to lock <0x00007f9ae7d7a6f0> (a java.lang.Integer)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:283)
        at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
        at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:65)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:878)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:54)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:29)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:307)
        at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:126)
        at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:120)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
        at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at org.mortbay.http.HttpServer.service(HttpServer.java:954)
        at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
        at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
        at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
        at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

"IPC Client connection to content-repo.cluster.powerset.com/XX.XX.XX.135:60000" daemon prio=10
tid=0x00007f9ade70b800 nid=0x5e47 runnable [0x00000000408d3000..0x00000000408d3d80]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)
        at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)
        - locked <0x00007f9ae90d42d0> (a sun.nio.ch.Util$1)
        - locked <0x00007f9ae90d42b8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00007f9ae90d3f28> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(Unknown Source)
        at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:237)
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:155)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:149)
        at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:122)
        at java.io.FilterInputStream.read(Unknown Source)
        at org.apache.hadoop.ipc.Client$Connection$1.read(Client.java:203)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        - locked <0x00007f9b4e8c1890> (a java.io.BufferedInputStream)
        at java.io.DataInputStream.readInt(Unknown Source)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:289)

"SocketListener0-1" prio=10 tid=0x00007f9ad0420000 nid=0x5879 in Object.wait() [0x0000000043746000..0x0000000043747c00]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.ipc.Client.call(Client.java:552)
        - locked <0x00007f9b48db8078> (a org.apache.hadoop.ipc.Client$Call)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java:230)
        at $Proxy1.findRootRegion(Unknown Source)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:799)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:464)
        - locked <0x00007f9ae7d7a6d8> (a java.lang.Integer)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:438)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:517)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:474)
        - locked <0x00007f9ae7d7a6f0> (a java.lang.Integer)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:446)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocation(HConnectionManager.java:283)
        at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:57)
        at org.apache.hadoop.hbase.client.ScannerCallable.instantiateServer(ScannerCallable.java:65)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:878)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:54)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:29)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:307)
        at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:126)
        at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:120)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
        at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at org.mortbay.http.HttpServer.service(HttpServer.java:954)
        at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
        at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
        at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
        at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

"SocketListener0-2" prio=10 tid=0x00007f9ad041f800 nid=0x5135 waiting on condition [0x0000000043241000..0x0000000043242c80]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:900)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:54)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:29)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:307)
        at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:126)
        at org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:120)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
        at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
        at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at org.mortbay.http.HttpServer.service(HttpServer.java:954)
        at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
        at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
        at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
        at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
        at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

"org.apache.hadoop.io.ObjectWritable Connection Culler" daemon prio=10 tid=0x00007f9ad0129000
nid=0x1bc4 waiting on condition [0x0000000042636000..0x0000000042636b00]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:435)

"IPC Server handler 9 on 60000" daemon prio=10 tid=0x00007f9ad813d800 nid=0x1bc3 waiting for
monitor entry [0x0000000042d3d000..0x0000000042d3da80]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

"IPC Server handler 9 on 60000" daemon prio=10 tid=0x00007f9ad813d800 nid=0x1bc3 waiting for
monitor entry [0x0000000042d3d000..0x0000000042d3da80]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

"IPC Server handler 8 on 60000" daemon prio=10 tid=0x00007f9ad8243800 nid=0x1bc2 waiting for
monitor entry [0x0000000042c3c000..0x0000000042c3ca00]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

"IPC Server handler 7 on 60000" daemon prio=10 tid=0x00007f9ad8242400 nid=0x1bc1 waiting for
monitor entry [0x0000000042b3b000..0x0000000042b3bd80]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

"IPC Server handler 6 on 60000" daemon prio=10 tid=0x00007f9ad8240c00 nid=0x1bc0 waiting for
monitor entry [0x0000000042a3a000..0x0000000042a3ad00]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

"IPC Server handler 5 on 60000" daemon prio=10 tid=0x00007f9ad823f800 nid=0x1bbf waiting for
monitor entry [0x0000000042939000..0x0000000042939c80]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

"IPC Server handler 4 on 60000" daemon prio=10 tid=0x00007f9ad823e800 nid=0x1bbe waiting for
monitor entry [0x0000000042838000..0x0000000042838c00]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)


"IPC Server handler 3 on 60000" daemon prio=10 tid=0x00007f9ad823d000 nid=0x1bbd waiting for
monitor entry [0x0000000042737000..0x0000000042737b80]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:173)
        - waiting to lock <0x00007f9ae7e96ab8> (a java.util.Collections$SynchronizedSortedMap)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:368)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:314)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:233)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:567)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

> locked up master
> ----------------
>
>                 Key: HBASE-870
>                 URL: https://issues.apache.org/jira/browse/HBASE-870
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> This morning, the pset master got into a locked up state.  Was stuck for tens of minutes
doing the below logging.  Cluster was unusable during this time:
> {code}
> ....
> 2008-09-04 18:06:45,893 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
REGION => {NAME => 'enwiki,YSWgGYWLLur87bjpYfj5--==,1220070599758', STARTKEY => 'YSWgGYWLLur87bjpYfj5--==',
ENDKEY => 'Y_dJHSiXE_hJ8jmGEgg1Dk==', ENCODED => 1060266767, TABLE => {{NAME =>
'enwiki', IS_ROOT => 'false'
> , IS_META => 'false', FAMILIES => [{NAME => 'alternate_title', BLOOMFILTER =>
'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647',
TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'anchor',
BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGT
> H => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
{NAME => 'alternate_url', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION
=> 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE
=> 'false'}, {NAME => 'page', BLOOMFILTER => 'false', VERSIONS => '21
> 47483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}, {NAME => 'misc', BLOOMFILTER => 'false', VERSIONS
=> '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY
=> 'false', BLOCKCACHE => 'false'}]}}}, SERVER => 'XX.XX.XX.1
> 83:60020', STARTCODE => 1219794634959
> 2008-09-04 18:06:45,895 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 1 time(s).
> 2008-09-04 18:06:46,908 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 2 time(s).
> 2008-09-04 18:06:47,918 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 3 time(s).
> 2008-09-04 18:06:48,928 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 4 time(s).
> 2008-09-04 18:06:49,938 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 5 time(s).
> 2008-09-04 18:06:50,958 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 6 time(s).
> 2008-09-04 18:06:51,978 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 7 time(s).
> 2008-09-04 18:06:52,398 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.76.44.68:60020}
> 2008-09-04 18:06:52,988 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 8 time(s).
> 2008-09-04 18:06:53,998 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 9 time(s).
> 2008-09-04 18:06:55,008 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 10 time(s).
> 2008-09-04 18:06:56,018 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: Connection refused
> 2008-09-04 18:06:56,045 DEBUG org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,
TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES =>
[{NAME => 'info', BLOOMFILTER => 'false', COMPRESSI
> ON => 'NONE', VERSIONS => '2147483647', LENGTH => '2147483647', TTL => '-1',
IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER =>
'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647',
TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '
> 208.76.44.48:60020', STARTCODE => 1219794657357
> 2008-09-04 18:07:01,084 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load:
1917, Num Servers: 69, Avg Load: 28.0
> 2008-09-04 18:07:01,084 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Cache hit in table locations for row <> and tableName .META.: location server XX.XX.XX.253:60020,
location region name .META.,,1
> 2008-09-04 18:07:01,087 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 1 time(s).
> 2008-09-04 18:07:02,088 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 2 time(s).
> 2008-09-04 18:07:03,108 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 3 time(s).
> 2008-09-04 18:07:04,118 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 4 time(s).
> 2008-09-04 18:07:05,138 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 5 time(s).
> 2008-09-04 18:07:06,038 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Removed .META.,,1 from cache because of shroomz_062108,a8XltuT_bfDh6RCYKWEf9-==,1215120916399
> 2008-09-04 18:07:06,158 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 6 time(s).
> 2008-09-04 18:07:07,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 7 time(s).
> 2008-09-04 18:07:08,178 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 8 time(s).
> 2008-09-04 18:07:09,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 9 time(s).
> 2008-09-04 18:07:10,208 INFO org.apache.hadoop.ipc.Client: Retrying connect to server:
aa0-004-19.u.powerset.com/XX.XX.XX.253:60020. Already tried 10 time(s).
> 2008-09-04 18:07:11,218 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: Connection refused
> 2008-09-04 18:08:09,779 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load:
1917, Num Servers: 69, Avg Load: 28.0
> 2008-09-04 18:08:11,238 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:09:11,248 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:09:43,328 INFO org.apache.hadoop.hbase.master.ServerManager: XX.XX.XX.185:60020
lease expired
> 2008-09-04 18:09:43,348 INFO org.apache.hadoop.hbase.master.ServerManager: XX.XX.XX.183:60020
lease expired
> 2008-09-04 18:09:43,368 INFO org.apache.hadoop.hbase.master.ServerManager: XX.XX.XX.231:60020
lease expired
> 2008-09-04 18:09:43,438 INFO org.apache.hadoop.hbase.master.ServerManager: XX.XX.XX.127:60020
lease expired
> 2008-09-04 18:09:43,528 INFO org.apache.hadoop.hbase.master.ServerManager: XX.XX.XX.68:60020
lease expired
> 2008-09-04 18:10:11,258 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:11:11,278 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:12:11,298 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:13:11,318 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:14:11,338 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:15:11,358 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:16:11,368 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:17:11,378 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:18:11,388 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:19:11,398 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:20:11,408 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:21:11,418 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:22:11,428 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:23:11,438 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> 2008-09-04 18:23:28,291 DEBUG org.apache.hadoop.hbase.master.ServerManager: Total Load:
1805, Num Servers: 65, Avg Load: 28.0
> 2008-09-04 18:24:11,458 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
reloading table servers because: timed out waiting for rpc response
> ....
> {code}
> Attached are thread dumps taking around this time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message