hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weihua JIANG <weihua.ji...@gmail.com>
Subject Re: Region not online after split by a closing RS
Date Tue, 05 Jul 2011 00:03:19 GMT
Both daughter regions are not online.

Thanks
Weihua

2011/7/4 Ted Yu <yuzhihong@gmail.com>:
> In the future, please direct questions on cdh releases to
> cdh-dev@cloudera.org
> You may cc dev@hbase.apache.org
>
> There is more than one minute difference between master and RS logs.
> Which one of the daughter regions didn't come online ?
>
> Cheers
>
> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <weihua.jiang@gmail.com> wrote:
>
>> The HBase version we are using is CDH3U0.
>>
>> Thanks
>> Weihua
>>
>> 2011/7/4 Weihua JIANG <weihua.jiang@gmail.com>:
>> > Hi all,
>> >
>> > We encountered a problem about region not onlining. A region is
>> > splitted by a closing RS and then this RS down. It seems master has
>> > known this split but it doesn't tried to make it online. Log from
>> > master
>> > 2011-06-30 22:58:52,945 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
>> > and split region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>> > checking daughter presence
>> > 2011-06-30 22:58:52,946 DEBUG
>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> > transition=RS_ZK_REGION_OPENING,
>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>> > region=ed60ec735e30db1d99290995eb1cd2d7
>> > 2011-06-30 22:58:53,005 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > present
>> > 2011-06-30 22:58:53,065 DEBUG
>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > present
>> >
>> > Log from RS is:
>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server handler 73 on 50820 caught:
>> > java.nio.channels.ClosedChannelException
>> >        at
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>> >        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>> >        at
>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>> >
>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> > Server handler 73 on 50820: exiting
>> > 2011-06-30 22:57:05,767 INFO
>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
>> > leases
>> > 2011-06-30 22:57:05,768 INFO
>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
>> > leases
>> > 2011-06-30 22:57:05,768 INFO
>> >
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> > Closed zookeeper sessionid=0x130ba69074900b4
>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>> > 0x130ba69074900b4 closed
>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
>> > EventThread shut down
>> > 2011-06-30 22:57:05,857 DEBUG
>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > 2011-06-30 22:57:05,863 DEBUG
>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > 2011-06-30 22:57:05,911 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
>> > in META
>> > 2011-06-30 22:57:05,942 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > in region .META.,,1, serverInfo=null
>> > 2011-06-30 22:57:05,943 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> > daughter
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>> > because stopping=false, stopped=true
>> > 2011-06-30 22:57:05,950 INFO
>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > in region .META.,,1, serverInfo=null
>> > 2011-06-30 22:57:05,950 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>> > daughter
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>> > because stopping=false, stopped=true
>> > 2011-06-30 22:57:06,004 INFO
>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
>> > updated, and report to master.
>> >
>> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
>> > new regions:
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
>> >
>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
>> > Split took 1mins, 12sec
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Split Thread to finish...
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Large Compaction Thread to finish...
>> > 2011-06-30 22:57:06,004 DEBUG
>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>> > Small Compaction Thread to finish...
>> > 2011-06-30 22:57:06,004 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
>> > exiting
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> > starting; hbase.shutdown.hook=true;
>> > fsShutdownHook=Thread[Thread-15,5,main]
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>> > hook
>> > 2011-06-30 22:57:06,090 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>> > shutdown hook thread.
>> > 2011-06-30 22:57:06,196 INFO
>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>> > finished.
>> >
>> >
>> > Thanks
>> > Weihua
>> >
>>
>

Mime
View raw message