hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Region not online after split by a closing RS
Date Mon, 04 Jul 2011 14:19:08 GMT
In the future, please direct questions on cdh releases to
cdh-dev@cloudera.org
You may cc dev@hbase.apache.org

There is more than one minute difference between master and RS logs.
Which one of the daughter regions didn't come online ?

Cheers

On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <weihua.jiang@gmail.com> wrote:

> The HBase version we are using is CDH3U0.
>
> Thanks
> Weihua
>
> 2011/7/4 Weihua JIANG <weihua.jiang@gmail.com>:
> > Hi all,
> >
> > We encountered a problem about region not onlining. A region is
> > splitted by a closing RS and then this RS down. It seems master has
> > known this split but it doesn't tried to make it online. Log from
> > master
> > 2011-06-30 22:58:52,945 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> > and split region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> > checking daughter presence
> > 2011-06-30 22:58:52,946 DEBUG
> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
> > transition=RS_ZK_REGION_OPENING,
> > server=hadoop01.sh.intel.com,50820,1309421825940,
> > region=ed60ec735e30db1d99290995eb1cd2d7
> > 2011-06-30 22:58:53,005 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > present
> > 2011-06-30 22:58:53,065 DEBUG
> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > present
> >
> > Log from RS is:
> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> > Server handler 73 on 50820 caught:
> > java.nio.channels.ClosedChannelException
> >        at
> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
> >        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
> >        at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
> >
> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> > Server handler 73 on 50820: exiting
> > 2011-06-30 22:57:05,767 INFO
> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
> > leases
> > 2011-06-30 22:57:05,768 INFO
> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
> > leases
> > 2011-06-30 22:57:05,768 INFO
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> > Closed zookeeper sessionid=0x130ba69074900b4
> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x130ba69074900b4 closed
> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
> > EventThread shut down
> > 2011-06-30 22:57:05,857 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > 2011-06-30 22:57:05,863 DEBUG
> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > 2011-06-30 22:57:05,911 INFO
> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
> > in META
> > 2011-06-30 22:57:05,942 INFO
> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > in region .META.,,1, serverInfo=null
> > 2011-06-30 22:57:05,943 INFO
> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> > daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> > because stopping=false, stopped=true
> > 2011-06-30 22:57:05,950 INFO
> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > in region .META.,,1, serverInfo=null
> > 2011-06-30 22:57:05,950 INFO
> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
> > daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> > because stopping=false, stopped=true
> > 2011-06-30 22:57:06,004 INFO
> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
> > updated, and report to master.
> >
> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
> > new regions:
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
> >
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
> > Split took 1mins, 12sec
> > 2011-06-30 22:57:06,004 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Split Thread to finish...
> > 2011-06-30 22:57:06,004 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Large Compaction Thread to finish...
> > 2011-06-30 22:57:06,004 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Small Compaction Thread to finish...
> > 2011-06-30 22:57:06,004 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
> > exiting
> > 2011-06-30 22:57:06,090 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> > starting; hbase.shutdown.hook=true;
> > fsShutdownHook=Thread[Thread-15,5,main]
> > 2011-06-30 22:57:06,090 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
> > hook
> > 2011-06-30 22:57:06,090 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
> > shutdown hook thread.
> > 2011-06-30 22:57:06,196 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> > finished.
> >
> >
> > Thanks
> > Weihua
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message