Return-Path: Delivered-To: apmail-hadoop-hbase-user-archive@minotaur.apache.org Received: (qmail 9943 invoked from network); 4 Aug 2009 22:23:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Aug 2009 22:23:20 -0000 Received: (qmail 12607 invoked by uid 500); 4 Aug 2009 22:23:24 -0000 Delivered-To: apmail-hadoop-hbase-user-archive@hadoop.apache.org Received: (qmail 12543 invoked by uid 500); 4 Aug 2009 22:23:24 -0000 Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-user@hadoop.apache.org Delivered-To: mailing list hbase-user@hadoop.apache.org Received: (qmail 12533 invoked by uid 99); 4 Aug 2009 22:23:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 22:23:24 +0000 X-ASF-Spam-Status: No, hits=3.7 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.221.188 as permitted sender) Received: from [209.85.221.188] (HELO mail-qy0-f188.google.com) (209.85.221.188) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 22:23:11 +0000 Received: by qyk26 with SMTP id 26so5371103qyk.5 for ; Tue, 04 Aug 2009 15:22:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to :content-type; bh=5jJQ+4Z921SBIZLjpbZ5wOjYmCPqZ5l87SKR+ylhG+s=; b=rant6k98hItV8CvEGGIkG3ObsbYyG68KJqgXWQky5C9R5tk5RMPj5u6Sp8ow5HZUcz QZrYbMmvWwzW2QXVlGwVE8qFyyRXZ1bRVNL9mESm2UBwqh8C8HYQCF2F5EuFBdpIhzCl 43FwucBFgRWuot0swHdP0MmRok5Uf2yevT+tM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; b=ijlOBeyW0Sm70U0P+jj3W4dFA+LBFYiEUZsS3bKKvn+115xH+1b4/vTMqczB9mjUpn sA3rres+k08Wi0jjhVCpp1EclPvkh29vNEU1ej09GFTs1zgLwfQLulkPxlyxD1+QtPz3 b2lbUMaiW7rBXvETrOyT41ItB84xbYmC0+lmE= MIME-Version: 1.0 Sender: saint.ack@gmail.com Received: by 10.229.110.5 with SMTP id l5mr2139594qcp.88.1249424569489; Tue, 04 Aug 2009 15:22:49 -0700 (PDT) In-Reply-To: References: Date: Tue, 4 Aug 2009 15:22:49 -0700 X-Google-Sender-Auth: 0b09702bbd2db646 Message-ID: <7c962aed0908041522u1d73bd13g2b7b48c41af57331@mail.gmail.com> Subject: Re: Region servers down when inserting with hbase0.20.0 rc From: stack To: hbase-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016364ef3d8a35bb20470585335 X-Virus-Checked: Checked by ClamAV on apache.org --0016364ef3d8a35bb20470585335 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit In the first, the RS lease in ZK expired so RS shut itself down (org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: Expired, type: None, path: null). Same in the second. Whats your ZK config. like? Try upping session timeout. Are nodes running on same nodes as hbase? Going by the below logs, you are seeing some long lags doing updates into hbase. What i/o looking like on these machines? Are the zk nodes being starved of i/o (if on same cluster)? St.Ack On Tue, Aug 4, 2009 at 1:23 AM, Zheng Lv wrote: > Hello Everyone, > We are testing hbase0.20.0 rc in a cluster with 5 nodes, inserting web > pages at a speed of 20 pages per second. A few minutes later one region > server shut down, and about 40 minutes later another shut down too, leaving > 2 of 4 region servers running. > I note that the contents of logs in two servers are not similar, and > below is the contents. > > ubuntu7: > > > 2009-08-04 08:59:35,870 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249344005673, > entries=34, > calcsize=10262, filesize=7353. New hlog > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 > 2009-08-04 09:12:49,462 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350000 to sun.nio.ch.SelectionKeyImpl@25cd0888 > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > lim=4 cap=4] > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:12:49,564 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: Disconnected, type: None, path: null > 2009-08-04 09:12:49,959 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu7/192.168.33.6:2222 > 2009-08-04 09:12:49,960 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.6:50860 remote=ubuntu7/192.168.33.6:2222] > 2009-08-04 09:12:49,960 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:12:49,965 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: SyncConnected, type: None, path: null > 2009-08-04 09:13:39,460 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350000 to sun.nio.ch.SelectionKeyImpl@3e472e76 > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > lim=4 cap=4] > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:13:39,567 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: Disconnected, type: None, path: null > 2009-08-04 09:13:39,807 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu6/192.168.33.7:2222 > 2009-08-04 09:13:45,807 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350000 to sun.nio.ch.SelectionKeyImpl@287b2e39 > java.io.IOException: TIMED OUT > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858) > 2009-08-04 09:13:45,807 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:13:46,192 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu9/192.168.33.5:2222 > 2009-08-04 09:13:52,184 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350000 to sun.nio.ch.SelectionKeyImpl@2f17b4f2 > java.io.IOException: TIMED OUT > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858) > 2009-08-04 09:13:52,185 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:13:52,923 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu3/192.168.33.8:2222 > 2009-08-04 09:13:55,338 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.6:49662 remote=ubuntu3/192.168.33.8:2222] > 2009-08-04 09:13:55,338 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:13:55,342 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350000 to sun.nio.ch.SelectionKeyImpl@3d689405 > java.io.IOException: Session Expired > at > > org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548) > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:13:55,342 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: Expired, type: None, path: null > 2009-08-04 09:13:55,343 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session > expired > 2009-08-04 09:13:55,343 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Restarting Region > Server > 2009-08-04 09:13:55,375 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > request=0.0, regions=1, stores=2, storefiles=2, storefileIndexSize=0, > memstoreSize=0, usedHeap=24, maxHeap=2993, blockCacheSize=5151568, > blockCacheFree=622696432, blockCacheCount=0, blockCacheHitRatio=0 > 2009-08-04 09:13:55,833 INFO > org.apache.hadoop.hbase.regionserver.LogFlusher: > regionserver/192.168.33.6:60020.logFlusher exiting > 2009-08-04 09:13:55,852 INFO > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > regionserver/192.168.33.6:60020.cacheFlusher exiting > 2009-08-04 09:13:55,882 INFO > org.apache.hadoop.hbase.regionserver.LogRoller: > LogRoller exiting. > 2009-08-04 09:13:58,339 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > 2009-08-04 09:13:58,339 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60020: exiting > 2009-08-04 09:13:58,340 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer > 2009-08-04 09:13:58,340 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC > Server Responder > 2009-08-04 09:13:58,340 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC > Server listener on 60020 > 2009-08-04 09:13:58,341 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60020: exiting > 2009-08-04 09:13:58,341 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60020: exiting > 2009-08-04 09:13:58,354 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60020: exiting > 2009-08-04 09:13:58,354 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60020: exiting > 2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60020: exiting > 2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60020: exiting > 2009-08-04 09:13:58,377 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60020: exiting > 2009-08-04 09:13:58,378 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60020: exiting > 2009-08-04 09:13:58,378 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60020: exiting > 2009-08-04 09:13:58,471 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > regionserver/192.168.33.6:60020.compactor exiting > 2009-08-04 09:13:58,472 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: > regionserver/192.168.33.6:60020.majorCompactionChecker exiting > 2009-08-04 09:14:00,187 INFO org.apache.hadoop.hbase.Leases: > regionserver/192.168.33.6:60020.leaseChecker closing leases > 2009-08-04 09:14:00,187 INFO org.apache.hadoop.hbase.Leases: > regionserver/192.168.33.6:60020.leaseChecker closed leases > 2009-08-04 09:14:01,895 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting > 2009-08-04 09:14:25,347 INFO org.apache.hadoop.hdfs.DFSClient: > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does > not exist. Holder DFSClient_1310853810 does not have any open files. > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at org.apache.hadoop.ipc.Client.call(Client.java:739) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.addBlock(Unknown Source) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > 2009-08-04 09:14:25,348 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for > replication for 26 seconds > 2009-08-04 09:14:25,348 WARN org.apache.hadoop.hdfs.DFSClient: > NotReplicatedYetException sleeping > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries > left > 4 > 2009-08-04 09:14:25,753 INFO org.apache.hadoop.hdfs.DFSClient: > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does > not exist. Holder DFSClient_1310853810 does not have any open files. > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at org.apache.hadoop.ipc.Client.call(Client.java:739) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.addBlock(Unknown Source) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > 2009-08-04 09:14:25,753 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for > replication for 27 seconds > 2009-08-04 09:14:25,753 WARN org.apache.hadoop.hdfs.DFSClient: > NotReplicatedYetException sleeping > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries > left > 3 > 2009-08-04 09:14:26,557 INFO org.apache.hadoop.hdfs.DFSClient: > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does > not exist. Holder DFSClient_1310853810 does not have any open files. > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at org.apache.hadoop.ipc.Client.call(Client.java:739) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.addBlock(Unknown Source) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > 2009-08-04 09:14:26,557 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for > replication for 28 seconds > 2009-08-04 09:14:26,557 WARN org.apache.hadoop.hdfs.DFSClient: > NotReplicatedYetException sleeping > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries > left > 2 > 2009-08-04 09:14:28,161 INFO org.apache.hadoop.hdfs.DFSClient: > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does > not exist. Holder DFSClient_1310853810 does not have any open files. > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at org.apache.hadoop.ipc.Client.call(Client.java:739) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.addBlock(Unknown Source) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > 2009-08-04 09:14:28,162 INFO org.apache.hadoop.hdfs.DFSClient: Waiting for > replication for 29 seconds > 2009-08-04 09:14:28,162 WARN org.apache.hadoop.hdfs.DFSClient: > NotReplicatedYetException sleeping > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 retries > left > 1 > 2009-08-04 09:14:31,367 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer > Exception: org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does > not exist. Holder DFSClient_1310853810 does not have any open files. > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at org.apache.hadoop.ipc.Client.call(Client.java:739) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at $Proxy1.addBlock(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy1.addBlock(Unknown Source) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2875) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2757) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2048) > at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2234) > 2009-08-04 09:14:31,367 WARN org.apache.hadoop.hdfs.DFSClient: Error > Recovery for block null bad datanode[0] nodes == null > 2009-08-04 09:14:31,368 WARN org.apache.hadoop.hdfs.DFSClient: Could not > get > block locations. Source file > "/hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858" - > Aborting... > 2009-08-04 09:14:31,371 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in > abort > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /hbase/.logs/ubuntu7,60020,1249344004988/hlog.dat.1249347575858 File does > not exist. Holder DFSClient_1310853810 does not have any open files. > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1317) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1308) > at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1236) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422) > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94) > at > > org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631) > at java.lang.Thread.run(Thread.java:619) > 2009-08-04 09:14:31,373 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed .META.,,1 > 2009-08-04 09:14:31,373 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at: > 192.168.33.6:60020 > 2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ZooKeeper: Closing > session: 0x22e2b69b350000 > 2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ClientCnxn: Closing > ClientCnxn for session: 0x22e2b69b350000 > 2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ClientCnxn: Disconnecting > ClientCnxn for session: 0x22e2b69b350000 > 2009-08-04 09:14:31,374 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x22e2b69b350000 closed > 2009-08-04 09:14:31,374 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/ > 192.168.33.6:60020 exiting > 2009-08-04 09:14:31,377 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: > Initializing RPC Metrics with hostName=HRegionServer, port=60020 > 2009-08-04 09:14:31,391 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown > thread. > 2009-08-04 09:14:31,392 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread > complete > 2009-08-04 09:14:31,394 INFO > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g > 2009-08-04 09:14:31,394 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms > 2009-08-04 09:14:31,397 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, > host=ubuntu9:2222,ubuntu7:2222,ubuntu3:2222,ubuntu2:2222,ubuntu6:2222 > sessionTimeout=30000 > watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@6a8c436b > 2009-08-04 09:14:31,398 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu3/192.168.33.8:2222 > 2009-08-04 09:14:31,401 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.6:49672 remote=ubuntu3/192.168.33.8:2222] > 2009-08-04 09:14:31,401 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:14:31,417 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: SyncConnected, type: None, path: null > 2009-08-04 09:14:31,447 INFO org.apache.zookeeper.ClientCnxn: EventThread > shut down > 2009-08-04 09:14:31,447 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > 2009-08-04 09:14:31,448 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer > 2009-08-04 09:14:31,452 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: telling master that > region server is shutting down at: 192.168.33.6:60020 > 2009-08-04 09:14:31,453 WARN > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed to send exiting > message to master: > java.lang.NullPointerException > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:660) > at java.lang.Thread.run(Thread.java:619) > 2009-08-04 09:14:31,453 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server at: > 192.168.33.6:60020 > 2009-08-04 09:14:31,453 INFO org.apache.zookeeper.ZooKeeper: Closing > session: 0x122e2b7548e0087 > 2009-08-04 09:14:31,453 INFO org.apache.zookeeper.ClientCnxn: Closing > ClientCnxn for session: 0x122e2b7548e0087 > 2009-08-04 09:14:31,460 INFO org.apache.zookeeper.ClientCnxn: Exception > while closing send thread for session 0x122e2b7548e0087 : Read error rc = > -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > > > > > > > > > ubuntu9: > > 2009-08-04 08:59:40,655 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,,1249347647788 > 2009-08-04 08:59:40,656 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,,1249347647788 > 2009-08-04 08:59:40,683 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,,1249347647788/491403556 available; sequence id is 0 > 2009-08-04 09:01:40,266 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249344130090, > entries=3147, calcsize=29823872, filesize=29491785. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254 > 2009-08-04 09:01:40,269 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249344130090 whose > highest sequence/edit id is 5984 > 2009-08-04 09:10:31,649 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254, > entries=6720, calcsize=63790080, filesize=63079139. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638 > 2009-08-04 09:11:30,345 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638, > entries=6715, calcsize=63794066, filesize=63090684. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324 > 2009-08-04 09:11:30,345 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249347700254 whose > highest sequence/edit id is 12705 > 2009-08-04 09:12:28,991 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324, > entries=6710, calcsize=63797225, filesize=63093218. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979 > 2009-08-04 09:12:28,991 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348231638 whose > highest sequence/edit id is 19419 > 2009-08-04 09:12:40,827 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,,1249347647788 > 2009-08-04 09:12:49,963 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region webpage,,1249347647788 in 9sec > 2009-08-04 09:13:29,072 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979, > entries=6720, calcsize=63800107, filesize=63095030. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060 > 2009-08-04 09:13:29,072 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348290324 whose > highest sequence/edit id is 26134 > 2009-08-04 09:14:27,438 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060, > entries=6715, calcsize=63798666, filesize=63094154. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426 > 2009-08-04 09:14:27,438 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348348979 whose > highest sequence/edit id is 32850 > 2009-08-04 09:14:56,116 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350001 to sun.nio.ch.SelectionKeyImpl@6db22920 > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > lim=4 cap=4] > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:14:57,075 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu3/192.168.33.8:2222 > 2009-08-04 09:14:57,076 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.5:44326 remote=ubuntu3/192.168.33.8:2222] > 2009-08-04 09:14:57,076 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:15:15,359 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,,1249347647788 > 2009-08-04 09:15:20,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region webpage,,1249347647788 in 5sec > 2009-08-04 09:15:20,972 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,,1249347647788 > 2009-08-04 09:15:21,963 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,,1249347647788 > 2009-08-04 09:15:24,791 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,,1249348520980/935635885 available; sequence id is 45742 > 2009-08-04 09:15:24,792 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,,1249348520980 > 2009-08-04 09:15:24,995 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980/1870783380 > available; sequence id is 45743 > 2009-08-04 09:15:24,996 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > 2009-08-04 09:15:25,039 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,,1249347647788', STARTKEY => '', ENDKEY => '', ENCODED => > 491403556, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'webpage', > FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => 'NONE', VERSIONS => '2', > TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE > => 'true'}, {NAME => 'CF_INFORMATION', COMPRESSION => 'NONE', VERSIONS => > '1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', > BLOCKCACHE => 'true'}]}}, new regions: webpage,,1249348520980, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980. > Split took 4sec > 2009-08-04 09:15:26,517 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,,1249348520980 > 2009-08-04 09:15:26,517 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > 2009-08-04 09:15:26,517 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,,1249348520980 > 2009-08-04 09:15:26,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,,1249348520980/935635885 available; sequence id is 45742 > 2009-08-04 09:15:26,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,,1249348520980 > 2009-08-04 09:15:26,690 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > 2009-08-04 09:15:26,998 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980/1870783380 > available; sequence id is 45743 > 2009-08-04 09:15:30,751 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region webpage,,1249348520980 in 4sec > 2009-08-04 09:15:30,751 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > 2009-08-04 09:16:00,015 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC > Server handler 0 on 60020 took 25134ms appending an edit to hlog; > editcount=6668 > 2009-08-04 09:16:00,016 WARN org.apache.hadoop.hbase.regionserver.HLog: > regionserver/192.168.33.5:60020.logFlusher took 19781ms optional sync'ing > hlog; editcount=6674 > 2009-08-04 09:16:00,397 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426, > entries=6710, calcsize=63779032, filesize=63075244. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377 > 2009-08-04 09:16:00,397 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348409060 whose > highest sequence/edit id is 39565 > 2009-08-04 09:16:30,001 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC > Server handler 1 on 60020 took 27826ms appending an edit to hlog; > editcount=204 > 2009-08-04 09:16:30,002 WARN org.apache.hadoop.hbase.regionserver.HLog: > regionserver/192.168.33.5:60020.logFlusher took 19986ms optional sync'ing > hlog; editcount=205 > 2009-08-04 09:16:38,451 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner > org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788 > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2255) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1871) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913) > 2009-08-04 09:16:38,457 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60020, call openScanner([B@47965ff3, startRow=, stopRow=, > maxVersions=1, timeRange=[0,9223372036854775807), > families={(family=CF_CONTENT, columns={}), (family=CF_INFORMATION, > columns={}}) from 192.168.33.7:52275: error: > org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788 > org.apache.hadoop.hbase.NotServingRegionException: webpage,,1249347647788 > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2255) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1871) > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913) > 2009-08-04 09:17:28,848 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377, > entries=6955, calcsize=66566006, filesize=65836364. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837 > 2009-08-04 09:17:28,848 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348467426 whose > highest sequence/edit id is 46277 > 2009-08-04 09:17:28,853 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348560377 whose > highest sequence/edit id is 53231 > 2009-08-04 09:17:29,981 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > in 1mins, 59sec > 2009-08-04 09:17:29,981 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > 2009-08-04 09:17:30,218 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980 > 2009-08-04 09:17:33,371 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989/1176865797 > available; sequence id is 53366 > 2009-08-04 09:17:33,372 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989 > 2009-08-04 09:17:33,575 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989/203332105 > available; sequence id is 53367 > 2009-08-04 09:17:33,576 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:17:34,580 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 0 time(s). > 2009-08-04 09:17:35,581 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 1 time(s). > 2009-08-04 09:17:36,582 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 2 time(s). > 2009-08-04 09:17:37,582 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 3 time(s). > 2009-08-04 09:17:38,583 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 4 time(s). > 2009-08-04 09:17:39,584 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 5 time(s). > 2009-08-04 09:17:40,585 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 6 time(s). > 2009-08-04 09:17:41,586 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 7 time(s). > 2009-08-04 09:17:42,586 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 8 time(s). > 2009-08-04 09:17:43,587 INFO org.apache.hadoop.ipc.HBaseClient: Retrying > connect to server: /192.168.33.6:60020. Already tried 9 time(s). > 2009-08-04 09:17:45,611 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348520980', > STARTKEY => 'http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744', > ENDKEY => '', ENCODED => 1870783380, OFFLINE => true, SPLIT => true, TABLE > => {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => > 'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', > COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989. > Split took 15sec > 2009-08-04 09:17:47,754 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989 > 2009-08-04 09:17:47,754 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:17:47,754 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989 > 2009-08-04 09:17:47,943 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989/1176865797 > available; sequence id is 53366 > 2009-08-04 09:17:47,944 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989 > 2009-08-04 09:17:47,944 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:17:48,163 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989/203332105 > available; sequence id is 53367 > 2009-08-04 09:17:52,777 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989 > in 4sec > 2009-08-04 09:17:52,778 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:18:01,313 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > in 8sec > 2009-08-04 09:24:26,812 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837, > entries=6661, calcsize=63769148, filesize=63059485. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799 > 2009-08-04 09:25:24,469 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799, > entries=6661, calcsize=63769500, filesize=63059617. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458 > 2009-08-04 09:25:24,469 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249348648837 whose > highest sequence/edit id is 59894 > 2009-08-04 09:25:34,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:25:38,992 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > in 4sec > 2009-08-04 09:25:38,992 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:25:39,244 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989 > 2009-08-04 09:25:42,458 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000/1497560335 > available; sequence id is 68239 > 2009-08-04 09:25:42,458 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000 > 2009-08-04 09:25:42,647 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000/1242271099 > available; sequence id is 68240 > 2009-08-04 09:25:42,647 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000 > 2009-08-04 09:25:42,663 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249348649989', > STARTKEY => 'http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168', > ENDKEY => '', ENCODED => 203332105, OFFLINE => true, SPLIT => true, TABLE > => > {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', COMPRESSION => > 'NONE', VERSIONS => '2', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', > COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => > '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348498311_4168,1249349139000, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348629221_6016,1249349139000. > Split took 3sec > 2009-08-04 09:30:24,918 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035 > 2009-08-04 09:30:24,918 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > 2009-08-04 09:30:24,918 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035 > 2009-08-04 09:30:25,058 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035/2028268999 > available; sequence id is 66248 > 2009-08-04 09:30:25,059 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035 > 2009-08-04 09:30:25,061 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > 2009-08-04 09:30:25,299 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035/1041724375 > available; sequence id is 66249 > 2009-08-04 09:30:28,717 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035 > in 3sec > 2009-08-04 09:30:28,717 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > 2009-08-04 09:30:41,191 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > in 12sec > 2009-08-04 09:31:14,443 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458, > entries=6662, calcsize=63774480, filesize=63064760. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432 > 2009-08-04 09:31:14,443 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349066799 whose > highest sequence/edit id is 66556 > 2009-08-04 09:32:12,213 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432, > entries=6661, calcsize=63776160, filesize=63066278. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174 > 2009-08-04 09:32:12,213 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349124458 whose > highest sequence/edit id is 73217 > 2009-08-04 09:32:35,692 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > 2009-08-04 09:32:38,318 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > in 2sec > 2009-08-04 09:32:38,318 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > 2009-08-04 09:32:38,479 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035 > 2009-08-04 09:32:40,510 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326/481931133 > available; sequence id is 82900 > 2009-08-04 09:32:42,063 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326 > 2009-08-04 09:32:42,256 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326/922277637 > available; sequence id is 82901 > 2009-08-04 09:32:42,256 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > 2009-08-04 09:32:42,298 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349460035', > STARTKEY => 'http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898', > ENDKEY => '', ENCODED => 1041724375, OFFLINE => true, SPLIT => true, TABLE > => {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => > '2', > COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY > => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS => > '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349219044_15898,1249349558326, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326. > Split took 3sec > 2009-08-04 09:32:43,169 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > 2009-08-04 09:32:43,170 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > 2009-08-04 09:32:43,315 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326/922277637 > available; sequence id is 82901 > 2009-08-04 09:32:43,315 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > 2009-08-04 09:32:55,285 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > in 11sec > 2009-08-04 09:32:55,285 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > 2009-08-04 09:32:55,912 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326 > 2009-08-04 09:32:56,630 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293/882105115 > available; sequence id is 83905 > 2009-08-04 09:32:56,631 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293 > 2009-08-04 09:32:56,793 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293/112267028 > available; sequence id is 83906 > 2009-08-04 09:32:56,794 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:32:56,808 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349558326', > STARTKEY => 'http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298', > ENDKEY => '', ENCODED => 922277637, OFFLINE => true, SPLIT => true, TABLE > => > {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2', > COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY > => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS => > '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293. > Split took 1sec > 2009-08-04 09:32:58,205 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:32:58,206 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293 > 2009-08-04 09:32:58,206 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:32:58,304 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293/112267028 > available; sequence id is 83906 > 2009-08-04 09:32:58,304 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293 > 2009-08-04 09:32:58,306 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:32:58,441 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293/882105115 > available; sequence id is 83905 > 2009-08-04 09:33:03,860 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > in 5sec > 2009-08-04 09:33:03,860 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293 > 2009-08-04 09:33:09,988 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293 > in 6sec > 2009-08-04 09:33:25,094 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174, > entries=6658, calcsize=63774356, filesize=63065350. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082 > 2009-08-04 09:33:25,094 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349474432 whose > highest sequence/edit id is 79878 > 2009-08-04 09:34:22,724 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082, > entries=6666, calcsize=63777964, filesize=63067534. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713 > 2009-08-04 09:34:22,725 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349532174 whose > highest sequence/edit id is 86545 > 2009-08-04 09:35:06,800 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:35:09,326 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > in 2sec > 2009-08-04 09:35:09,326 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:35:09,561 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293 > 2009-08-04 09:35:12,780 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334/316888594 > available; sequence id is 98604 > 2009-08-04 09:35:12,781 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334 > 2009-08-04 09:35:12,949 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334/1414734782 > available; sequence id is 98605 > 2009-08-04 09:35:12,950 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334 > 2009-08-04 09:35:12,966 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349575293', > STARTKEY => 'http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720', > ENDKEY => '', ENCODED => 112267028, OFFLINE => true, SPLIT => true, TABLE > => > {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2', > COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY > => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS => > '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349500641_20720,1249349709334, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349709334. > Split took 3sec > 2009-08-04 09:37:58,723 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622 > 2009-08-04 09:37:58,724 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622 > 2009-08-04 09:38:18,205 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622/1005978655 > available; sequence id is 113675 > 2009-08-04 09:38:18,206 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622 > 2009-08-04 09:38:22,211 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622 > in 4sec > 2009-08-04 09:47:32,557 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732 > 2009-08-04 09:47:32,557 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > 2009-08-04 09:47:32,557 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732 > 2009-08-04 09:47:32,694 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732/671938285 > available; sequence id is 174598 > 2009-08-04 09:47:32,694 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > 2009-08-04 09:47:32,694 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732 > 2009-08-04 09:47:32,913 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732/469485305 > available; sequence id is 174599 > 2009-08-04 09:47:35,243 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732 > in 2sec > 2009-08-04 09:47:35,243 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > 2009-08-04 09:47:46,699 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > in 11sec > 2009-08-04 09:47:48,978 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713, > entries=6662, calcsize=63776160, filesize=63066462. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966 > 2009-08-04 09:47:48,978 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349605082 whose > highest sequence/edit id is 93207 > 2009-08-04 09:48:46,548 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966, > entries=6661, calcsize=63776160, filesize=63066278. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350526537 > 2009-08-04 09:48:46,548 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249349662713 whose > highest sequence/edit id is 175865 > 2009-08-04 09:49:42,602 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > 2009-08-04 09:49:44,186 INFO org.apache.hadoop.hbase.regionserver.HLog: > Roll > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350526537, > entries=6661, calcsize=63776160, filesize=63066258. New hlog > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350584168 > 2009-08-04 09:49:44,187 INFO org.apache.hadoop.hbase.regionserver.HLog: > removing old hlog file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350468966 whose > highest sequence/edit id is 182525 > 2009-08-04 09:49:48,938 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > in 6sec > 2009-08-04 09:49:48,938 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting split of region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > 2009-08-04 09:49:49,199 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732 > 2009-08-04 09:49:49,954 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945/2137368991 > available; sequence id is 189657 > 2009-08-04 09:49:49,955 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945 > 2009-08-04 09:49:50,115 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945/904839552 > available; sequence id is 189658 > 2009-08-04 09:49:50,115 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945 > 2009-08-04 09:49:50,167 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META > updated, and report to master all successful. Old region=REGION => {NAME => > 'webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350537732', > STARTKEY => 'http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046', > ENDKEY => '', ENCODED => 469485305, OFFLINE => true, SPLIT => true, TABLE > => > {{NAME => 'webpage', FAMILIES => [{NAME => 'CF_CONTENT', VERSIONS => '2', > COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY > => 'false', BLOCKCACHE => 'true'}, {NAME => 'CF_INFORMATION', VERSIONS => > '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', > IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945, > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350402915_36232,1249350588945. > Split took 1sec > 2009-08-04 09:49:50,799 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945 > 2009-08-04 09:49:50,799 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945 > 2009-08-04 09:49:50,933 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945/2137368991 > available; sequence id is 189657 > 2009-08-04 09:49:50,934 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945 > 2009-08-04 09:49:55,807 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945 > in 4sec > 2009-08-04 09:54:51,308 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427 > 2009-08-04 09:54:51,309 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: > MSG_REGION_OPEN: > webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427 > 2009-08-04 09:54:51,486 INFO org.apache.hadoop.hbase.regionserver.HRegion: > region webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427/2108149192 > available; sequence id is 220779 > 2009-08-04 09:54:51,487 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Starting compaction on region webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427 > 2009-08-04 09:54:53,885 INFO org.apache.hadoop.hbase.regionserver.HRegion: > compaction completed on region > webpage,http:\x2F\x2Fnews.163.com > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427 > in 2sec > 2009-08-04 09:56:14,418 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x322e2b5f53a0000 to sun.nio.ch.SelectionKeyImpl@2345f0e3 > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > lim=4 cap=4] > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:56:14,519 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: Disconnected, type: None, path: null > 2009-08-04 09:56:15,314 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu7/192.168.33.6:2222 > 2009-08-04 09:56:18,137 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350001 to sun.nio.ch.SelectionKeyImpl@49271218 > java.io.IOException: TIMED OUT > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858) > 2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu9/192.168.33.5:2222 > 2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.5:43793 remote=ubuntu9/192.168.33.5:2222] > 2009-08-04 09:56:19,033 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:56:19,035 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350001 to sun.nio.ch.SelectionKeyImpl@cf78c3d > java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 > lim=4 cap=4] > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:56:19,036 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown input > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:56:19,036 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:56:19,192 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu7/192.168.33.6:2222 > 2009-08-04 09:56:21,307 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x322e2b5f53a0000 to sun.nio.ch.SelectionKeyImpl@3c9d926a > java.io.IOException: TIMED OUT > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858) > 2009-08-04 09:56:21,307 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:56:21,436 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu6/192.168.33.7:2222 > 2009-08-04 09:56:25,187 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x22e2b69b350001 to sun.nio.ch.SelectionKeyImpl@5017ff71 > java.io.IOException: TIMED OUT > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858) > 2009-08-04 09:56:25,187 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:56:25,316 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu6/192.168.33.7:2222 > 2009-08-04 09:56:27,437 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x322e2b5f53a0000 to sun.nio.ch.SelectionKeyImpl@57837ccb > java.io.IOException: TIMED OUT > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:858) > 2009-08-04 09:56:27,438 WARN org.apache.zookeeper.ClientCnxn: Ignoring > exception during shutdown output > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651) > at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922) > 2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu3/192.168.33.8:2222 > 2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.5:54934 remote=ubuntu3/192.168.33.8:2222] > 2009-08-04 09:56:27,667 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:56:27,670 WARN org.apache.zookeeper.ClientCnxn: Exception > closing session 0x322e2b5f53a0000 to sun.nio.ch.SelectionKeyImpl@7c3edbed > java.io.IOException: Session Expired > at > > org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:548) > at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:661) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897) > 2009-08-04 09:56:27,671 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, > state: Expired, type: None, path: null > 2009-08-04 09:56:27,671 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: ZooKeeper session > expired > 2009-08-04 09:56:27,671 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Restarting Region > Server > 2009-08-04 09:56:27,674 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: > request=0.0, regions=8, stores=16, storefiles=16, storefileIndexSize=1, > memstoreSize=0, usedHeap=202, maxHeap=2993, blockCacheSize=97949616, > blockCacheFree=529898384, blockCacheCount=843, blockCacheHitRatio=5 > 2009-08-04 09:56:28,307 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.5:47976 remote=ubuntu6/192.168.33.7:2222] > 2009-08-04 09:56:28,307 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > 2009-08-04 09:56:29,527 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > server on 60020 > 2009-08-04 09:56:29,528 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 9 on 60020: exiting > 2009-08-04 09:56:29,528 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 6 on 60020: exiting > 2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 8 on 60020: exiting > 2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 0 on 60020: exiting > 2009-08-04 09:56:29,529 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 1 on 60020: exiting > 2009-08-04 09:56:29,529 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer > 2009-08-04 09:56:29,539 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 4 on 60020: exiting > 2009-08-04 09:56:29,541 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC > Server Responder > 2009-08-04 09:56:29,542 INFO org.apache.hadoop.ipc.HBaseServer: Stopping > IPC > Server listener on 60020 > 2009-08-04 09:56:29,543 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 7 on 60020: exiting > 2009-08-04 09:56:29,575 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 5 on 60020: exiting > 2009-08-04 09:56:29,576 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 3 on 60020: exiting > 2009-08-04 09:56:29,576 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60020: exiting > 2009-08-04 09:56:29,681 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: > regionserver/192.168.33.5:60020.compactor exiting > 2009-08-04 09:56:29,681 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer$MajorCompactionChecker: > regionserver/192.168.33.5:60020.majorCompactionChecker exiting > 2009-08-04 09:56:29,682 INFO > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > regionserver/192.168.33.5:60020.cacheFlusher exiting > 2009-08-04 09:56:29,682 INFO > org.apache.hadoop.hbase.regionserver.LogFlusher: > regionserver/192.168.33.5:60020.logFlusher exiting > 2009-08-04 09:56:29,683 INFO > org.apache.hadoop.hbase.regionserver.LogRoller: > LogRoller exiting. > 2009-08-04 09:56:29,694 ERROR > org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to close log in > abort > java.io.IOException: java.io.IOException: Could not complete write to file > /hbase/.logs/ubuntu9,60020,1249344129415/hlog.dat.1249350584168 by > DFSClient_1458021990 > at > org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:449) > at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94) > at > > org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:631) > at java.lang.Thread.run(Thread.java:619) > 2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350280477_34046,1249350588945 > 2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348370059_1744,1249348649989 > 2009-08-04 09:56:29,696 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249348939079_11112,1249349460035 > 2009-08-04 09:56:29,697 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,,1249348520980 > 2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350669622_41010,1249350974427 > 2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349361130_18298,1249349575293 > 2009-08-04 09:56:29,698 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249349568563_22014,1249349965622 > 2009-08-04 09:56:29,699 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed webpage,http:\x2F\x2Fnews.163.com > > \x2F09\x2F0803\x2F01\x2F5FOO155J0001124J.html1249350166325_32162,1249350537732 > 2009-08-04 09:56:29,699 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at: > 192.168.33.5:60020 > 2009-08-04 09:56:30,501 INFO org.apache.hadoop.hbase.Leases: > regionserver/192.168.33.5:60020.leaseChecker closing leases > 2009-08-04 09:56:30,501 INFO org.apache.hadoop.hbase.Leases: > regionserver/192.168.33.5:60020.leaseChecker closed leases > 2009-08-04 09:56:31,488 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting > 2009-08-04 09:56:31,488 INFO org.apache.zookeeper.ZooKeeper: Closing > session: 0x322e2b5f53a0000 > 2009-08-04 09:56:31,488 INFO org.apache.zookeeper.ClientCnxn: Closing > ClientCnxn for session: 0x322e2b5f53a0000 > 2009-08-04 09:56:31,489 INFO org.apache.zookeeper.ClientCnxn: Disconnecting > ClientCnxn for session: 0x322e2b5f53a0000 > 2009-08-04 09:56:31,489 INFO org.apache.zookeeper.ZooKeeper: Session: > 0x322e2b5f53a0000 closed > 2009-08-04 09:56:31,489 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/ > 192.168.33.5:60020 exiting > 2009-08-04 09:56:31,490 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Starting shutdown > thread. > 2009-08-04 09:56:31,491 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Shutdown thread > complete > 2009-08-04 09:56:31,499 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: > Initializing RPC Metrics with hostName=HRegionServer, port=60020 > 2009-08-04 09:56:31,500 INFO > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: > globalMemStoreLimit=1.2g, globalMemStoreLimitLowMark=748.5m, maxHeap=2.9g > 2009-08-04 09:56:31,500 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms > 2009-08-04 09:56:31,503 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, > host=ubuntu9:2222,ubuntu7:2222,ubuntu3:2222,ubuntu2:2222,ubuntu6:2222 > sessionTimeout=30000 > watcher=org.apache.hadoop.hbase.regionserver.HRegionServer@68da4b71 > 2009-08-04 09:56:31,504 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server ubuntu6/192.168.33.7:2222 > 2009-08-04 09:56:31,519 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 192.168.33.5:47982 remote=ubuntu6/192.168.33.7:2222] > 2009-08-04 09:56:31,519 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > > > > > > > > contents in hbase-site.xml: > > > hbase.rootdir > hdfs://ubuntu6:9000/hbase > The directory shared by region servers. > > > > hbase.cluster.distributed > true > The mode the cluster will be in. Possible values are > false: standalone and pseudo-distributed setups with managed Zookeeper > true: fully-distributed with unmanaged Zookeeper Quorum (see > hbase-env.sh) > > > > hbase.zookeeper.property.clientPort > 2222 > Property from ZooKeeper's config zoo.cfg. > The port at which the clients will connect. > > > > hbase.zookeeper.quorum > ubuntu2,ubuntu3,ubuntu7,ubuntu9,ubuntu6 > Comma separated list of servers in the ZooKeeper Quorum. > For example, "host1.mydomain.com,host2.mydomain.com, > host3.mydomain.com > ". > By default this is set to localhost for local and pseudo-distributed > modes > of operation. For a fully-distributed setup, this should be set to a > full > list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in > hbase-env.sh > this is the list of servers which we will start/stop ZooKeeper on. > > > > Any help? > > Thanks a lot, > LvZheng > --0016364ef3d8a35bb20470585335--