hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weihua JIANG <weihua.ji...@gmail.com>
Subject Re: Region not online after split by a closing RS
Date Mon, 04 Jul 2011 12:30:20 GMT
The HBase version we are using is CDH3U0.

Thanks
Weihua

2011/7/4 Weihua JIANG <weihua.jiang@gmail.com>:
> Hi all,
>
> We encountered a problem about region not onlining. A region is
> splitted by a closing RS and then this RS down. It seems master has
> known this split but it doesn't tried to make it online. Log from
> master
> 2011-06-30 22:58:52,945 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> and split region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> checking daughter presence
> 2011-06-30 22:58:52,946 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop01.sh.intel.com,50820,1309421825940,
> region=ed60ec735e30db1d99290995eb1cd2d7
> 2011-06-30 22:58:53,005 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> present
> 2011-06-30 22:58:53,065 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> present
>
> Log from RS is:
> 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 73 on 50820 caught:
> java.nio.channels.ClosedChannelException
>        at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>        at org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>
> 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server handler 73 on 50820: exiting
> 2011-06-30 22:57:05,767 INFO
> org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
> leases
> 2011-06-30 22:57:05,768 INFO
> org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
> leases
> 2011-06-30 22:57:05,768 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Closed zookeeper sessionid=0x130ba69074900b4
> 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x130ba69074900b4 closed
> 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
> EventThread shut down
> 2011-06-30 22:57:05,857 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> 2011-06-30 22:57:05,863 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> 2011-06-30 22:57:05,911 INFO
> org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
> in META
> 2011-06-30 22:57:05,942 INFO
> org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> in region .META.,,1, serverInfo=null
> 2011-06-30 22:57:05,943 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> daughter CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> because stopping=false, stopped=true
> 2011-06-30 22:57:05,950 INFO
> org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> in region .META.,,1, serverInfo=null
> 2011-06-30 22:57:05,950 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> daughter CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> because stopping=false, stopped=true
> 2011-06-30 22:57:06,004 INFO
> org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
> updated, and report to master.
> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
> new regions: CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
> Split took 1mins, 12sec
> 2011-06-30 22:57:06,004 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Split Thread to finish...
> 2011-06-30 22:57:06,004 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Large Compaction Thread to finish...
> 2011-06-30 22:57:06,004 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Small Compaction Thread to finish...
> 2011-06-30 22:57:06,004 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
> exiting
> 2011-06-30 22:57:06,090 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> starting; hbase.shutdown.hook=true;
> fsShutdownHook=Thread[Thread-15,5,main]
> 2011-06-30 22:57:06,090 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> hook
> 2011-06-30 22:57:06,090 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> shutdown hook thread.
> 2011-06-30 22:57:06,196 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> finished.
>
>
> Thanks
> Weihua
>

Mime
View raw message