hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Lv <lvzheng19800...@gmail.com>
Subject master kills itself
Date Sun, 16 Aug 2009 09:49:23 GMT
Hello,
    Thank you for your suggestions.
    Several days before We found our routing talbe has some problems, after
adjusting now we are sure that the bandwidth is ok.
    And we have used lzo compression.
    So we started the test program again, but after running normally for 23
hours, the master killed itself. Following is part of the log.
    By the way, this time we inserted 10 webpages per second only.
2009-08-14 13:36:31,840 INFO org.apache.hadoop.hbase.master.ServerManager: 4
region servers, 0 dead, average load 48.75
2009-08-14 13:36:32,016 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 192.168.33.5:60020,
regionnam
e: .META.,,1, startKey: <>}
2009-08-14 13:36:32,076 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 192.168.33.6:60020,
regionnam
e: -ROOT-,,0, startKey: <>}
2009-08-14 13:36:32,084 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server:
192.168.33.6:60020
, regionname: -ROOT-,,0, startKey: <>} complete
2009-08-14 13:36:32,316 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 193 row(s) of meta region {server:
192.168.33.5:600
20, regionname: .META.,,1, startKey: <>} complete
2009-08-14 13:36:32,316 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2009-08-14 13:37:00,366 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80001 to sun.nio.ch.SelectionKeyImpl@4a407c9f
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-14 13:37:00,881 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-14 13:37:04,366 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80000 to sun.nio.ch.SelectionKeyImpl@4ac6ee33
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-14 13:37:04,721 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-08-14 13:37:08,872 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80001 to sun.nio.ch.SelectionKeyImpl@2e93ebe0
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-14 13:37:08,873 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
        at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
        at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-14 13:37:09,486 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu2/192.168.33.9:2222
2009-08-14 13:37:12,712 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80000 to sun.nio.ch.SelectionKeyImpl@7162d703
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-14 13:37:12,713 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
        at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
        at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-14 13:37:13,032 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu3/192.168.33.8:2222
2009-08-14 13:37:17,482 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80001 to sun.nio.ch.SelectionKeyImpl@1012401d
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-14 13:37:17,483 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
        at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
        at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-14 13:37:17,856 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-14 13:37:19,445 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.7:40923 remote
=ubuntu7/192.168.33.6:2222]
2009-08-14 13:37:19,445 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-14 13:37:21,022 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80000 to sun.nio.ch.SelectionKeyImpl@2e101b3a
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
2009-08-14 13:37:21,023 WARN org.apache.zookeeper.ClientCnxn: Ignoring
exception during shutdown output
java.net.SocketException: Transport endpoint is not connected
        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
        at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
        at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
        at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
2009-08-14 13:37:21,908 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ubuntu7/192.168.33.6:2222
2009-08-14 13:37:21,908 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
192.168.33.7:40926 remote
=ubuntu7/192.168.33.6:2222]
2009-08-14 13:37:21,909 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2009-08-14 13:37:21,911 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x22313002be80000 to sun.nio.ch.SelectionKeyImpl@6bdfe124
java.io.IOException: Session Expired
        at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
        at
org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
2009-08-14 13:37:21,912 ERROR org.apache.hadoop.hbase.master.HMaster: Master
lost its znode, killing itself now
Regards,
LvZheng

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message