hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: is there any problem with our environment?
Date Wed, 30 Sep 2009 03:22:40 GMT
Are you using IPs?  Try using hostnames?
St.Ack

2009/9/27 Zheng Lv <lvzheng19800619@gmail.com>

> following is the logs in another one of shutdown regionservers:
>
>
> 2009年 09月 27日 星期日 15:40:20 CST Starting regionserver on ubuntu7
> ulimit -n 1024
> 2009-09-27 15:40:20,951 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> vmInputArguments=[-Xmx3000m, -Djava.net.preferIPv4Stack=true,
> -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC,
> -XX:+CMSIncrementalMode, -Dhbase.log.dir=/opt/hadoop/hbase/bin/../logs,
> -Dhbase.log.file=hbase-cyd-regionserver-ubuntu7.log,
> -Dhbase.home.dir=/opt/hadoop/hbase/bin/.., -Dhbase.id.str=cyd,
> -Dhbase.root.logger=INFO,DRFA,
> -Djava.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64]
> 2009-09-27 15:40:20,961 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: My address is
> ubuntu7:60020
> 2009-09-27 15:40:21,003 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
> Initializing RPC Metrics with hostName=HRegionServer, port=60020
> 2009-09-27 15:40:21,011 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g
> 2009-09-27 15:40:21,013 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.2.0--1, built on 05/15/2009 06:05 GMT
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:host.name=ubuntu7
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_14
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.14/jre
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
>
> environment:java.class.path=/home/cyd/conf/hbase:/usr/lib/jvm/java-6-sun/lib/tools.jar:/opt/hadoop/hbase/bin/..:/opt/hadoop/hbase/bin/../hbase-0.20.0.jar:/opt/hadoop/hbase/bin/../lib/AgileJSON-2009-03-30.jar:/opt/hadoop/hbase/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/hadoop/hbase/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/hadoop/hbase/bin/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/hbase/bin/../lib/commons-math-1.1.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-plus4681-core.jar:/opt/hadoop/hbase/bin/../lib/hadoop-0.20.0-test.jar:/opt/hadoop/hbase/bin/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/hbase/bin/../lib/jetty-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jetty-util-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/jruby-complete-1.2.0.jar:/opt/hadoop/hbase/bin/../lib/json.jar:/opt/hadoop/hbase/bin/../lib/junit-3.8.1.jar:/opt/hadoop/hbase/bin/../lib/libthrift-r771587.jar:/opt/hadoop/hbase/bin/../lib/log4j-1.2.15.jar:/opt/hadoop/hbase/bin/../lib/lucene-core-2.2.0.jar:/opt/hadoop/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hadoop/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hadoop/hbase/bin/../lib/zookeeper-r785019-hbase-1329.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/hbase/bin/../lib/jsp-2.1/jsp-api-2.1.jar
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
>
> environment:java.library.path=/opt/hadoop/hbase/bin/../lib/native/Linux-amd64-64
> 2009-09-27 15:40:21,027 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.name=Linux
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:os.version=2.6.28-11-server
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.name=cyd
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.home=/home/cyd
> 2009-09-27 15:40:21,028 INFO org.apache.zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/hadoop/hbase-0.20.0
> 2009-09-27 15:40:21,029 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, host=ubuntu12:2222,ubuntu10:2222,ubuntu2:2222
> sessionTimeout=240000
> watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@6719dc16
> 2009-09-27 15:40:21,030 INFO org.apache.zookeeper.ClientCnxn:
> zookeeper.disableAutoWatchReset is false
> 2009-09-27 15:40:21,042 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 15:40:21,048 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.6:36316 remote=ubuntu10/192.168.33.10:2222]
> 2009-09-27 15:40:21,060 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 15:40:21,138 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: SyncConnected, type: None, path: null
> 2009-09-27 15:40:21,362 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to read master
> address from ZooKeeper. Retrying. Error was:
> java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
>  at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:331)
>  at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:240)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1316)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1348)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:428)
>  at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/master
>  at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
>  at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>  at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:750)
>  at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:327)
>  ... 5 more
> 2009-09-27 15:40:21,455 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: SyncConnected, type: NodeCreated, path: /hbase/master
> 2009-09-27 15:40:21,469 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
> 192.168.33.7:60000 that we are up
> 2009-09-27 15:40:24,368 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
> 192.168.33.7:60000 that we are up
> 2009-09-27 15:41:03,732 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us
> address
> to use. Was=192.168.33.6:60020, Now=192.168.33.6
> 2009-09-27 15:41:04,267 INFO org.apache.hadoop.hbase.regionserver.HLog:
> HLog
> configuration: blocksize=67108864, rollsize=63753420, enabled=true,
> flushlogentries=100, optionallogflushinternal=10000ms
> 2009-09-27 15:41:04,468 INFO org.apache.hadoop.hbase.regionserver.HLog: New
> hlog /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267
> 2009-09-27 15:41:04,477 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=RegionServer,
> sessionId=regionserver/192.168.33.6:60020
> 2009-09-27 15:41:04,478 INFO
> org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics:
> Initialized
> 2009-09-27 15:41:04,818 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> to port 60030
> 2009-09-27 15:41:05,167 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> Responder: starting
> 2009-09-27 15:41:05,167 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> listener on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: starting
> 2009-09-27 15:41:05,168 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: starting
> 2009-09-27 15:41:05,169 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: starting
> 2009-09-27 15:41:05,169 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started
> at: 192.168.33.6:60020
> 2009-09-27 15:41:05,198 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile:
> Allocating LruBlockCache with maximum size 598.8m
> 2009-09-27 15:41:08,229 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
> .META.,,1
> 2009-09-27 15:41:08,237 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
> MSG_REGION_OPEN:
> .META.,,1
> 2009-09-27 15:41:08,333 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> region .META.,,1/1028785192 available; sequence id is 0
> 2009-09-27 15:50:48,238 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
> limited
> 2009-09-27 16:19:49,375 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@7745c859
> java.io.IOException: TIMED OUT
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:19:49,476 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-09-27 16:19:50,182 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:19:53,188 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@16e334a
> java.net.NoRouteToHostException: No route to host
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:19:53,189 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>  at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:53,189 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:53,919 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:19:56,018 INFO org.apache.hadoop.ipc.Client: Retrying connect
> to server: ubuntu6/192.168.33.7:9000. Already tried 0 time(s).
> 2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@506937f9
> java.net.NoRouteToHostException: No route to host
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>  at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:56,928 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:19:58,605 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:20:01,617 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@6e75d758
> java.net.NoRouteToHostException: No route to host
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:20:01,618 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>  at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:01,618 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:02,323 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:20:05,328 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@7a8d0b0f
> java.net.NoRouteToHostException: No route to host
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:20:05,328 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>  at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:05,329 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:05,879 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:20:08,887 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@41d144a4
> java.net.NoRouteToHostException: No route to host
>  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:885)
> 2009-09-27 16:20:08,888 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown input
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
>  at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:08,889 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.nio.channels.ClosedChannelException
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:20:10,535 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:20:13,527 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.6:60330 remote=ubuntu10/192.168.33.10:2222]
> 2009-09-27 16:20:13,528 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:20:13,532 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: SyncConnected, type: None, path: null
> 2009-09-27 16:21:33,555 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@11f18cbe
> java.io.IOException: TIMED OUT
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:21:33,656 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Disconnected, type: None, path: null
> 2009-09-27 16:21:33,713 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu12/192.168.33.12:2222
> 2009-09-27 16:21:47,047 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@66e8c7db
> java.io.IOException: TIMED OUT
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:21:47,048 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
>  at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:21:47,227 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu2/192.168.33.9:2222
> 2009-09-27 16:22:00,566 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@6d0718b7
> java.io.IOException: TIMED OUT
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858)
> 2009-09-27 16:22:00,567 WARN org.apache.zookeeper.ClientCnxn: Ignoring
> exception during shutdown output
> java.net.SocketException: Transport endpoint is not connected
>  at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>  at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
>  at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
>  at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
> 2009-09-27 16:22:02,264 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 0)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,271 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 1)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,276 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 2)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,281 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 3)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,286 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 4)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,290 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 5)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,294 INFO org.apache.zookeeper.ClientCnxn: Attempting
> connection to server ubuntu10/192.168.33.10:2222
> 2009-09-27 16:22:02,295 INFO org.apache.zookeeper.ClientCnxn: Priming
> connection to java.nio.channels.SocketChannel[connected local=/
> 192.168.33.6:60343 remote=ubuntu10/192.168.33.10:2222]
> 2009-09-27 16:22:02,295 INFO org.apache.zookeeper.ClientCnxn: Server
> connection successful
> 2009-09-27 16:22:02,296 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 6)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,298 WARN org.apache.zookeeper.ClientCnxn: Exception
> closing session 0x123fa71ce2e0000 to sun.nio.ch.SelectionKeyImpl@23bf8cd5
> java.io.IOException: Session Expired
>  at
>
> org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548)
>  at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661)
>  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
> 2009-09-27 16:22:02,298 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event,
> state: Expired, type: None, path: null
> 2009-09-27 16:22:02,299 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session
> expired
> 2009-09-27 16:22:02,301 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 7)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,303 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> request=0.0, regions=1, stores=2, storefiles=0, storefileIndexSize=0,
> memstoreSize=0, usedHeap=29, maxHeap=2993, blockCacheSize=5151568,
> blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0
> 2009-09-27 16:22:02,307 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message
> (Retry: 8)
> org.apache.hadoop.hbase.Leases$LeaseStillHeldException
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:02,307 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stop was requested,
> clearing the toDo despite of the exception
> 2009-09-27 16:22:02,308 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 60020
> 2009-09-27 16:22:02,308 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 0 on 60020: exiting
> 2009-09-27 16:22:02,309 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 1 on 60020: exiting
> 2009-09-27 16:22:02,309 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 3 on 60020: exiting
> 2009-09-27 16:22:02,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server listener on 60020
> 2009-09-27 16:22:02,310 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> IPC
> Server Responder
> 2009-09-27 16:22:02,310 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
> 2009-09-27 16:22:02,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 4 on 60020: exiting
> 2009-09-27 16:22:02,325 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 5 on 60020: exiting
> 2009-09-27 16:22:02,328 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 6 on 60020: exiting
> 2009-09-27 16:22:02,329 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 7 on 60020: exiting
> 2009-09-27 16:22:02,329 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 8 on 60020: exiting
> 2009-09-27 16:22:02,330 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 2 on 60020: exiting
> 2009-09-27 16:22:02,330 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
> handler 9 on 60020: exiting
> 2009-09-27 16:22:02,465 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
> regionserver/192.168.33.6:60020.cacheFlusher exiting
> 2009-09-27 16:22:02,465 INFO
> org.apache.hadoop.hbase.regionserver.LogFlusher:
> regionserver/192.168.33.6:60020.logFlusher exiting
> 2009-09-27 16:22:02,465 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller:
> LogRoller exiting.
> 2009-09-27 16:22:02,466 INFO
> org.apache.hadoop.hbase.regionserver.CompactSplitThread:
> regionserver/192.168.33.6:60020.compactor exiting
> 2009-09-27 16:22:02,466 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker:
> regionserver/192.168.33.6:60020.majorCompactionChecker exiting
> 2009-09-27 16:22:02,478 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
>  at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>  at org.apache.hadoop.ipc.Client.call(Client.java:739)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>  at $Proxy1.addBlock(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>  at $Proxy1.addBlock(Unknown Source)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:02,478 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 4
> 2009-09-27 16:22:02,883 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
>  at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>  at org.apache.hadoop.ipc.Client.call(Client.java:739)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>  at $Proxy1.addBlock(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>  at $Proxy1.addBlock(Unknown Source)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:02,883 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 3
> 2009-09-27 16:22:03,687 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
>  at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>  at org.apache.hadoop.ipc.Client.call(Client.java:739)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>  at $Proxy1.addBlock(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>  at $Proxy1.addBlock(Unknown Source)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:03,687 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 2
> 2009-09-27 16:22:04,520 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.6:60020.leaseChecker closing leases
> 2009-09-27 16:22:04,521 INFO org.apache.hadoop.hbase.Leases:
> regionserver/192.168.33.6:60020.leaseChecker closed leases
> 2009-09-27 16:22:05,292 INFO org.apache.hadoop.hdfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
>  at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>  at org.apache.hadoop.ipc.Client.call(Client.java:739)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>  at $Proxy1.addBlock(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>  at $Proxy1.addBlock(Unknown Source)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:05,292 WARN org.apache.hadoop.hdfs.DFSClient:
> NotReplicatedYetException sleeping
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 retries
> left
> 1
> 2009-09-27 16:22:08,370 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting
> 2009-09-27 16:22:08,497 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
>  at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>  at org.apache.hadoop.ipc.Client.call(Client.java:739)
>  at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>  at $Proxy1.addBlock(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>  at
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>  at $Proxy1.addBlock(Unknown Source)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048)
>  at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234)
> 2009-09-27 16:22:08,498 WARN org.apache.hadoop.hdfs.DFSClient: Error
> Recovery for block null bad datanode[0] nodes == null
> 2009-09-27 16:22:08,498 WARN org.apache.hadoop.hdfs.DFSClient: Could not
> get
> block locations. Source file
> "/hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267" -
> Aborting...
> 2009-09-27 16:22:08,499 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in
> abort
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException:
> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
> /hbase/.logs/ubuntu7,60020,1254037263642/hlog.dat.1254037264267 File does
> not exist. Holder DFSClient_679922832 does not have any open files.
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308)
>  at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236)
>  at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>  at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>  at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:396)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94)
>  at
>
> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48)
>  at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:637)
>  at java.lang.Thread.run(Thread.java:619)
> 2009-09-27 16:22:08,501 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed .META.,,1
> 2009-09-27 16:22:08,502 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at:
> 192.168.33.6:60020
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ZooKeeper: Closing
> session: 0x123fa71ce2e0000
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ClientCnxn: Closing
> ClientCnxn for session: 0x123fa71ce2e0000
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
> ClientCnxn for session: 0x123fa71ce2e0000
> 2009-09-27 16:22:08,502 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x123fa71ce2e0000 closed
> 2009-09-27 16:22:08,503 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2009-09-27 16:22:08,505 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/
> 192.168.33.6:60020 exiting
> 2009-09-27 16:22:08,507 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown
> thread.
> 2009-09-27 16:22:08,507 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread
> complete
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message