hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Weihua JIANG <weihua.ji...@gmail.com>
Subject Re: Region not online after split by a closing RS
Date Tue, 05 Jul 2011 02:10:57 GMT
According to RS log, it seems it performs a region split during this
RS shutdown process. Thus, RS only performs the split and add the two
daughter regions to META, but not open them (MetaEditor.addDaughter()
with serverInfo as null). It seems it's the correct behavior for RS.

But, I can't find the corresponding code on master side to assign
these unassigned regions. And the log on master side doesn't show an
hint of such assignment decision.

Thanks
Weihua

2011/7/5 Weihua JIANG <weihua.jiang@gmail.com>:
> Both daughter regions are offline.
>
> The full log of master related to this RS down are:
>
> 2011-06-30 22:56:51,621 INFO
> org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
> ephemeral node deleted, processing expiration
> [xmlqa-clv4.sh.intel.com,50820,1309421874765]
> 2011-06-30 22:56:51,622 DEBUG
> org.apache.hadoop.hbase.master.ServerManager:
> Added=xmlqa-clv4.sh.intel.com,50820,1309421874765 to dead servers,
> submitted shutdown handler to be executed, root=false, meta=false
> 2011-06-30 22:56:51,622 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Splitting logs for xmlqa-clv4.sh.intel.com,50820,1309421874765
> 2011-06-30 22:56:51,633 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting 35
> hlog(s) in hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
> 2011-06-30 22:56:51,633 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> Thread[WriterThread-0,5,main]: starting
> 2011-06-30 22:56:51,633 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> Thread[WriterThread-1,5,main]: starting
> 2011-06-30 22:56:51,633 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 1 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944,
> length=63870691
> 2011-06-30 22:56:51,634 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:56:51,634 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Writer thread
> Thread[WriterThread-2,5,main]: starting
> 2011-06-30 22:56:52,636 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:56:52,736 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,736 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
> region=ed60ec735e30db1d99290995eb1cd2d7
> 2011-06-30 22:56:52,777 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,777 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
> region=625da13f7c9f8d2a6db3e7bd892de061
> 2011-06-30 22:56:52,790 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,790 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
> region=6a22ac864ca20a7f34b8822b6b092697
> 2011-06-30 22:56:52,825 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,826 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
> region=0ea99d0d995b95e8ea07441d30aa5545
> 2011-06-30 22:56:52,836 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,836 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
> region=32c8d35a95c6ac4f399c17dfa04066b1
> 2011-06-30 22:56:52,841 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,841 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
> region=d5a18e344d77bdccb568fd59623fb566
> 2011-06-30 22:56:52,868 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,868 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
> region=e4a2320bc8ff30fee1a9b20267b399cf
> 2011-06-30 22:56:52,909 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:52,909 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
> region=de5cb72653d016804cbd16f4a71470cd
> 2011-06-30 22:56:53,017 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,018 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
> region=5a4a923e81c6127eb7f3ed61e5774063
> 2011-06-30 22:56:53,023 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,024 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
> region=3e0c81217d4b47920e11960f1816efae
> 2011-06-30 22:56:53,069 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,069 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
> region=d673806c5348d8517dcdb3392e5f5b21
> 2011-06-30 22:56:53,101 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,101 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
> region=7cfa5c93f2d46d01c8f60ff934053ad1
> 2011-06-30 22:56:53,109 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,109 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
> region=84c6f6ce13ffbeb3eb83c765fef754f3
> 2011-06-30 22:56:53,179 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,179 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
> region=c94dc395183fd5a2286ec823b194b517
> 2011-06-30 22:56:53,190 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs -- HDFS-200
> 2011-06-30 22:56:53,190 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer
> path=hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
> region=0dd9c11896eca61505861a9e49f2fef3
> 2011-06-30 22:56:54,198 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25752
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:56:54,198 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 2 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636,
> length=73429706
> 2011-06-30 22:56:54,198 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:56:55,201 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:56:57,787 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=29756
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:56:57,787 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 3 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577,
> length=65063774
> 2011-06-30 22:56:57,788 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:56:58,793 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:57:00,533 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26415
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:57:00,533 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 4 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792,
> length=64631565
> 2011-06-30 22:57:00,533 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:57:01,537 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:57:03,755 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25941
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:57:03,755 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 5 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002,
> length=68367834
> 2011-06-30 22:57:03,755 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:57:04,757 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:57:06,463 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27679
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:57:06,463 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 6 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699,
> length=64503295
> 2011-06-30 22:57:06,463 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:57:07,465 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:57:09,089 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26171
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:57:09,089 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 7 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029,
> length=64201147
> 2011-06-30 22:57:09,089 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:57:10,093 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:57:11,792 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25904
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:57:11,792 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 8 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992,
> length=63800579
> 2011-06-30 22:57:11,792 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:57:12,794 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:57:18,504 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:57:18,504 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 9 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571,
> length=64119848
> 2011-06-30 22:57:18,504 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:57:19,506 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:57:21,160 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26023
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:57:21,160 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 10 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271,
> length=64643463
> 2011-06-30 22:57:21,160 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:57:22,162 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:57:23,541 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
> currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
> dead server
> 2011-06-30 22:57:23,573 WARN org.apache.hadoop.ipc.HBaseServer: IPC
> Server listener on 50800: readAndProcess threw exception
> java.io.IOException: Connection reset by peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
>        at sun.nio.ch.FileDispatcher.read0(Native Method)
>        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
>        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
>        at sun.nio.ch.IOUtil.read(IOUtil.java:206)
>        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>        at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1357)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:900)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:522)
>        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:316)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:619)
> 2011-06-30 22:57:23,674 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26324
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:57:23,674 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 11 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420,
> length=64232645
> 2011-06-30 22:57:23,674 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:57:23,734 DEBUG
> org.apache.hadoop.hbase.master.ServerManager: Server REPORT rejected;
> currently processing xmlqa-clv4.sh.intel.com,50820,1309421874765 as
> dead server
> 2011-06-30 22:57:24,678 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:57:26,287 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25901
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:57:26,288 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 12 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336,
> length=68916014
> 2011-06-30 22:57:26,288 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:57:27,291 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:57:29,540 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=27997
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:57:29,540 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 13 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618,
> length=64023210
> 2011-06-30 22:57:29,541 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:57:30,543 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:57:32,118 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25852
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:57:32,118 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 14 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430,
> length=63963259
> 2011-06-30 22:57:32,118 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:57:33,123 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:57:36,027 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25800
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:57:36,027 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 15 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319,
> length=65390175
> 2011-06-30 22:57:36,028 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:57:37,031 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:57:38,618 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26591
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:57:38,618 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 16 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600,
> length=64082403
> 2011-06-30 22:57:38,618 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:57:39,623 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:57:42,129 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26079
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:57:42,129 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 17 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088,
> length=64438939
> 2011-06-30 22:57:42,129 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:57:43,132 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:57:44,965 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26012
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:57:44,965 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 18 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332,
> length=64295447
> 2011-06-30 22:57:44,965 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:57:45,968 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:57:48,826 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26038
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:57:48,826 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 19 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952,
> length=64592159
> 2011-06-30 22:57:48,826 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:57:49,828 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:57:53,219 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26061
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:57:53,219 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 20 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863,
> length=64057243
> 2011-06-30 22:57:53,220 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:57:54,224 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:57:58,220 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25765
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:57:58,220 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 21 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965,
> length=64306648
> 2011-06-30 22:57:58,220 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:57:59,225 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:58:00,572 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26006
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:58:00,572 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 22 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999,
> length=64238208
> 2011-06-30 22:58:00,572 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:01,577 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:04,010 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25961
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:04,010 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 23 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242,
> length=66022861
> 2011-06-30 22:58:04,011 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:05,017 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:08,583 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26599
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:08,583 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 24 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462,
> length=64341194
> 2011-06-30 22:58:08,583 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:09,585 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:11,171 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25978
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:11,171 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 25 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064,
> length=64083581
> 2011-06-30 22:58:11,171 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:12,173 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:14,655 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26044
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:14,656 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 26 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989,
> length=64093785
> 2011-06-30 22:58:14,656 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:15,658 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:17,914 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25712
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:17,915 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 27 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493,
> length=64115777
> 2011-06-30 22:58:17,915 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:18,918 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:20,762 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25760
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:20,762 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 28 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468,
> length=64628500
> 2011-06-30 22:58:20,762 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:21,766 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:23,158 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26131
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:23,159 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 29 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048,
> length=64522869
> 2011-06-30 22:58:23,159 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:24,162 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:25,593 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26159
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:25,593 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 30 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226,
> length=64127204
> 2011-06-30 22:58:25,593 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:26,594 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:28,023 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:28,023 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 31 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208,
> length=96405599
> 2011-06-30 22:58:28,023 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:29,026 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:31,603 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38922
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:31,603 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 32 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723,
> length=64219650
> 2011-06-30 22:58:31,603 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:32,607 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:33,924 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=25927
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:33,924 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 33 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548,
> length=64734596
> 2011-06-30 22:58:33,924 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:34,927 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:37,455 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=26330
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:37,456 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 34 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138,
> length=95982267
> 2011-06-30 22:58:37,456 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:38,459 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:40,216 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=38754
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:40,216 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog
> 35 of 35: hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661,
> length=0
> 2011-06-30 22:58:40,216 INFO org.apache.hadoop.hbase.util.FSUtils:
> Recovering file
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:41,219 INFO org.apache.hadoop.hbase.util.FSUtils:
> Finished lease recover attempt for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:41,219 WARN
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> might be still open, length is 0
> 2011-06-30 22:58:42,146 DEBUG
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Pushed=23462
> entries from hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:42,179 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444065944
> 2011-06-30 22:58:42,185 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444166636
> 2011-06-30 22:58:42,191 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444228577
> 2011-06-30 22:58:42,196 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444255792
> 2011-06-30 22:58:42,203 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444335002
> 2011-06-30 22:58:42,255 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444371699
> 2011-06-30 22:58:42,270 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444412029
> 2011-06-30 22:58:42,276 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444443992
> 2011-06-30 22:58:42,292 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444493571
> 2011-06-30 22:58:42,300 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444532271
> 2011-06-30 22:58:42,306 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444550420
> 2011-06-30 22:58:42,321 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444576336
> 2011-06-30 22:58:42,329 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444612618
> 2011-06-30 22:58:42,336 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444647430
> 2011-06-30 22:58:42,342 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444662319
> 2011-06-30 22:58:42,348 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444724600
> 2011-06-30 22:58:42,354 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444755088
> 2011-06-30 22:58:42,360 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444815332
> 2011-06-30 22:58:42,366 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444851952
> 2011-06-30 22:58:42,372 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444922863
> 2011-06-30 22:58:42,378 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309444966965
> 2011-06-30 22:58:42,408 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445006999
> 2011-06-30 22:58:42,414 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445053242
> 2011-06-30 22:58:42,420 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445102462
> 2011-06-30 22:58:42,426 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445159064
> 2011-06-30 22:58:42,437 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445179989
> 2011-06-30 22:58:42,444 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445212493
> 2011-06-30 22:58:42,491 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445244468
> 2011-06-30 22:58:42,504 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445279048
> 2011-06-30 22:58:42,510 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445347226
> 2011-06-30 22:58:42,516 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445379208
> 2011-06-30 22:58:42,522 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445610723
> 2011-06-30 22:58:42,528 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445661548
> 2011-06-30 22:58:42,534 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445709138
> 2011-06-30 22:58:42,540 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
> processed log hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> to hdfs://namenode.hadoop.intel.com/hbase/.oldlogs/xmlqa-clv4.sh.intel.com%3A50820.1309445733661
> 2011-06-30 22:58:42,546 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting for
> split writer threads to finish
> 2011-06-30 22:58:49,942 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split writers
> finished
> 2011-06-30 22:58:50,002 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0dd9c11896eca61505861a9e49f2fef3/recovered.edits/0000000000019932905
> (wrote 62511 edits in 6859ms)
> 2011-06-30 22:58:50,014 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/0ea99d0d995b95e8ea07441d30aa5545/recovered.edits/0000000000019932894
> (wrote 63273 edits in 9130ms)
> 2011-06-30 22:58:51,226 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/32c8d35a95c6ac4f399c17dfa04066b1/recovered.edits/0000000000019932946
> (wrote 63193 edits in 29376ms)
> 2011-06-30 22:58:51,232 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/3e0c81217d4b47920e11960f1816efae/recovered.edits/0000000000019932904
> (wrote 62727 edits in 16595ms)
> 2011-06-30 22:58:52,413 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/5a4a923e81c6127eb7f3ed61e5774063/recovered.edits/0000000000019932893
> (wrote 62672 edits in 7177ms)
> 2011-06-30 22:58:52,461 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/625da13f7c9f8d2a6db3e7bd892de061/recovered.edits/0000000000019932885
> (wrote 62636 edits in 52110ms)
> 2011-06-30 22:58:52,509 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/6a22ac864ca20a7f34b8822b6b092697/recovered.edits/0000000000019932886
> (wrote 62860 edits in 19402ms)
> 2011-06-30 22:58:52,515 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/7cfa5c93f2d46d01c8f60ff934053ad1/recovered.edits/0000000000019933928
> (wrote 62718 edits in 12592ms)
> 2011-06-30 22:58:52,605 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/84c6f6ce13ffbeb3eb83c765fef754f3/recovered.edits/0000000000019934632
> (wrote 62561 edits in 12576ms)
> 2011-06-30 22:58:52,670 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/c94dc395183fd5a2286ec823b194b517/recovered.edits/0000000000019932895
> (wrote 62860 edits in 9056ms)
> 2011-06-30 22:58:52,719 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d5a18e344d77bdccb568fd59623fb566/recovered.edits/0000000000019932901
> (wrote 63328 edits in 8381ms)
> 2011-06-30 22:58:52,725 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/d673806c5348d8517dcdb3392e5f5b21/recovered.edits/0000000000019933162
> (wrote 62040 edits in 7900ms)
> 2011-06-30 22:58:52,731 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/de5cb72653d016804cbd16f4a71470cd/recovered.edits/0000000000019932902
> (wrote 62397 edits in 19988ms)
> 2011-06-30 22:58:52,779 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/e4a2320bc8ff30fee1a9b20267b399cf/recovered.edits/0000000000019932888
> (wrote 62952 edits in 21391ms)
> 2011-06-30 22:58:52,785 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path
> hdfs://namenode.hadoop.intel.com/hbase/CMCC_Detail_ReversePhoneMonth__DateCat_NONE/ed60ec735e30db1d99290995eb1cd2d7/recovered.edits/0000000000019932884
> (wrote 62765 edits in 15781ms)
> 2011-06-30 22:58:52,785 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
> splitting completed in 121163 ms for
> hdfs://namenode.hadoop.intel.com/hbase/.logs/xmlqa-clv4.sh.intel.com,50820,1309421874765
> 2011-06-30 22:58:52,893 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
> Reassigning 16 region(s) that
> xmlqa-clv4.sh.intel.com,50820,1309421874765 was carrying (skipping 0
> regions(s) that are already in transition)
> 2011-06-30 22:58:52,894 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 6a22ac864ca20a7f34b8822b6b092697 with OFFLINE state
> 2011-06-30 22:58:52,896 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.,
> src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,896 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,,1309422002872.6a22ac864ca20a7f34b8822b6b092697.
> to hadoop05.sh.intel.com,50820,1309421836726
> 2011-06-30 22:58:52,898 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 625da13f7c9f8d2a6db3e7bd892de061 with OFFLINE state
> 2011-06-30 22:58:52,900 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.,
> src=, dest=harpertown083.sh.intel.com,50820,1309421853805; 10
> (online=10, exclude=null) available servers
> 2011-06-30 22:58:52,900 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,06666666666666660,1309422002873.625da13f7c9f8d2a6db3e7bd892de061.
> to harpertown083.sh.intel.com,50820,1309421853805
> 2011-06-30 22:58:52,900 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop05.sh.intel.com,50820,1309421836726,
> region=6a22ac864ca20a7f34b8822b6b092697
> 2011-06-30 22:58:52,902 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 32c8d35a95c6ac4f399c17dfa04066b1 with OFFLINE state
> 2011-06-30 22:58:52,904 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.,
> src=, dest=xmlqa-clv6.sh.intel.com,50820,1309421880784; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,904 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,13333333333333320,1309422002874.32c8d35a95c6ac4f399c17dfa04066b1.
> to xmlqa-clv6.sh.intel.com,50820,1309421880784
> 2011-06-30 22:58:52,904 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=harpertown083.sh.intel.com,50820,1309421853805,
> region=625da13f7c9f8d2a6db3e7bd892de061
> 2011-06-30 22:58:52,906 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 0ea99d0d995b95e8ea07441d30aa5545 with OFFLINE state
> 2011-06-30 22:58:52,935 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.,
> src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,935 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,19999999999999980,1309422002875.0ea99d0d995b95e8ea07441d30aa5545.
> to hadoop05.sh.intel.com,50820,1309421836726
> 2011-06-30 22:58:52,936 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=xmlqa-clv6.sh.intel.com,50820,1309421880784,
> region=32c8d35a95c6ac4f399c17dfa04066b1
> 2011-06-30 22:58:52,937 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for d5a18e344d77bdccb568fd59623fb566 with OFFLINE state
> 2011-06-30 22:58:52,939 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.,
> src=, dest=hbase01.sh.intel.com,50820,1309421885586; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,939 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,26666666666666640,1309422002876.d5a18e344d77bdccb568fd59623fb566.
> to hbase01.sh.intel.com,50820,1309421885586
> 2011-06-30 22:58:52,940 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop05.sh.intel.com,50820,1309421836726,
> region=0ea99d0d995b95e8ea07441d30aa5545
> 2011-06-30 22:58:52,941 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for ed60ec735e30db1d99290995eb1cd2d7 with OFFLINE state
> 2011-06-30 22:58:52,943 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hbase01.sh.intel.com,50820,1309421885586,
> region=d5a18e344d77bdccb568fd59623fb566
> 2011-06-30 22:58:52,943 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.,
> src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:52,943 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,33333333333333300,1309422002877.ed60ec735e30db1d99290995eb1cd2d7.
> to hadoop01.sh.intel.com,50820,1309421825940
> 2011-06-30 22:58:52,945 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
> and split region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
> checking daughter presence
> 2011-06-30 22:58:52,946 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop01.sh.intel.com,50820,1309421825940,
> region=ed60ec735e30db1d99290995eb1cd2d7
> 2011-06-30 22:58:53,005 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
> present
> 2011-06-30 22:58:53,065 DEBUG
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
> present
> 2011-06-30 22:58:53,065 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 3e0c81217d4b47920e11960f1816efae with OFFLINE state
> 2011-06-30 22:58:53,067 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.,
> src=, dest=hadoop01.sh.intel.com,50820,1309421825940; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,067 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,46666666666666620,1309422002878.3e0c81217d4b47920e11960f1816efae.
> to hadoop01.sh.intel.com,50820,1309421825940
> 2011-06-30 22:58:53,070 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for e4a2320bc8ff30fee1a9b20267b399cf with OFFLINE state
> 2011-06-30 22:58:53,072 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop01.sh.intel.com,50820,1309421825940,
> region=3e0c81217d4b47920e11960f1816efae
> 2011-06-30 22:58:53,072 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.,
> src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,072 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,53333333333333280,1309422002879.e4a2320bc8ff30fee1a9b20267b399cf.
> to hadoop06.sh.intel.com,50820,1309421846110
> 2011-06-30 22:58:53,083 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 5a4a923e81c6127eb7f3ed61e5774063 with OFFLINE state
> 2011-06-30 22:58:53,085 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.,
> src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,085 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,59999999999999940,1309422002880.5a4a923e81c6127eb7f3ed61e5774063.
> to ditest02.sh.intel.com,50820,1309421803646
> 2011-06-30 22:58:53,085 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop06.sh.intel.com,50820,1309421846110,
> region=e4a2320bc8ff30fee1a9b20267b399cf
> 2011-06-30 22:58:53,086 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for d673806c5348d8517dcdb3392e5f5b21 with OFFLINE state
> 2011-06-30 22:58:53,087 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.,
> src=, dest=hadoop06.sh.intel.com,50820,1309421846110; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,087 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,66666666666666600,1309422002881.d673806c5348d8517dcdb3392e5f5b21.
> to hadoop06.sh.intel.com,50820,1309421846110
> 2011-06-30 22:58:53,088 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest02.sh.intel.com,50820,1309421803646,
> region=5a4a923e81c6127eb7f3ed61e5774063
> 2011-06-30 22:58:53,090 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 7cfa5c93f2d46d01c8f60ff934053ad1 with OFFLINE state
> 2011-06-30 22:58:53,117 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.,
> src=, dest=hadoop05.sh.intel.com,50820,1309421836726; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,117 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,73333333333333260,1309422002882.7cfa5c93f2d46d01c8f60ff934053ad1.
> to hadoop05.sh.intel.com,50820,1309421836726
> 2011-06-30 22:58:53,117 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop06.sh.intel.com,50820,1309421846110,
> region=d673806c5348d8517dcdb3392e5f5b21
> 2011-06-30 22:58:53,120 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 84c6f6ce13ffbeb3eb83c765fef754f3 with OFFLINE state
> 2011-06-30 22:58:53,126 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.,
> src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,126 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,79999999999999920,1309422002882.84c6f6ce13ffbeb3eb83c765fef754f3.
> to ditest02.sh.intel.com,50820,1309421803646
> 2011-06-30 22:58:53,127 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=hadoop05.sh.intel.com,50820,1309421836726,
> region=7cfa5c93f2d46d01c8f60ff934053ad1
> 2011-06-30 22:58:53,129 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for c94dc395183fd5a2286ec823b194b517 with OFFLINE state
> 2011-06-30 22:58:53,143 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.,
> src=, dest=ditest01.sh.intel.com,50820,1309421799506; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,143 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,86666666666666580,1309422002883.c94dc395183fd5a2286ec823b194b517.
> to ditest01.sh.intel.com,50820,1309421799506
> 2011-06-30 22:58:53,143 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest02.sh.intel.com,50820,1309421803646,
> region=84c6f6ce13ffbeb3eb83c765fef754f3
> 2011-06-30 22:58:53,149 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest01.sh.intel.com,50820,1309421799506,
> region=c94dc395183fd5a2286ec823b194b517
> 2011-06-30 22:58:53,151 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for 0dd9c11896eca61505861a9e49f2fef3 with OFFLINE state
> 2011-06-30 22:58:53,153 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.,
> src=, dest=ditest02.sh.intel.com,50820,1309421803646; 10 (online=10,
> exclude=null) available servers
> 2011-06-30 22:58:53,153 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,93333333333333240,1309422002884.0dd9c11896eca61505861a9e49f2fef3.
> to ditest02.sh.intel.com,50820,1309421803646
> 2011-06-30 22:58:53,156 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZKAssign:
> master:50800-0x230ba68644c0066 Creating (or updating) unassigned node
> for d9adac25d8b83a3d4c252acaf5ba17af with OFFLINE state
> 2011-06-30 22:58:53,157 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: No previous
> transition plan was found (or we are ignoring an existing plan) for
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
> so generated a random one;
> hri=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.,
> src=, dest=harpertown0816.sh.intel.com,50820,1309421864349; 10
> (online=10, exclude=null) available servers
> 2011-06-30 22:58:53,157 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,99999999999999900,1309422002885.d9adac25d8b83a3d4c252acaf5ba17af.
> to harpertown0816.sh.intel.com,50820,1309421864349
> 2011-06-30 22:58:53,158 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_OPENING,
> server=ditest02.sh.intel.com,50820,1309421803646,
> region=0dd9c11896eca61505861a9e49f2fef3
> 2011-06-30 22:58:53,159 INFO
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished
> processing of shutdown of xmlqa-clv4.sh.intel.com,50820,1309421874765
>
>
> Thanks
> Weihua
>
> 2011/7/4 Ted Yu <yuzhihong@gmail.com>:
>> In the future, please direct questions on cdh releases to
>> cdh-dev@cloudera.org
>> You may cc dev@hbase.apache.org
>>
>> There is more than one minute difference between master and RS logs.
>> Which one of the daughter regions didn't come online ?
>>
>> Cheers
>>
>> On Mon, Jul 4, 2011 at 5:30 AM, Weihua JIANG <weihua.jiang@gmail.com> wrote:
>>
>>> The HBase version we are using is CDH3U0.
>>>
>>> Thanks
>>> Weihua
>>>
>>> 2011/7/4 Weihua JIANG <weihua.jiang@gmail.com>:
>>> > Hi all,
>>> >
>>> > We encountered a problem about region not onlining. A region is
>>> > splitted by a closing RS and then this RS down. It seems master has
>>> > known this split but it doesn't tried to make it online. Log from
>>> > master
>>> > 2011-06-30 22:58:52,945 DEBUG
>>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined
>>> > and split region
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.;
>>> > checking daughter presence
>>> > 2011-06-30 22:58:52,946 DEBUG
>>> > org.apache.hadoop.hbase.master.AssignmentManager: Handling
>>> > transition=RS_ZK_REGION_OPENING,
>>> > server=hadoop01.sh.intel.com,50820,1309421825940,
>>> > region=ed60ec735e30db1d99290995eb1cd2d7
>>> > 2011-06-30 22:58:53,005 DEBUG
>>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > present
>>> > 2011-06-30 22:58:53,065 DEBUG
>>> > org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > present
>>> >
>>> > Log from RS is:
>>> > 2011-06-30 22:57:05,207 WARN org.apache.hadoop.ipc.HBaseServer: IPC
>>> > Server handler 73 on 50820 caught:
>>> > java.nio.channels.ClosedChannelException
>>> >        at
>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
>>> >        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>>> >        at
>>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java:1342)
>>> >        at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBaseServer.java:727)
>>> >        at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServer.java:792)
>>> >        at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1083)
>>> >
>>> > 2011-06-30 22:57:05,207 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> > Server handler 73 on 50820: exiting
>>> > 2011-06-30 22:57:05,767 INFO
>>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closing
>>> > leases
>>> > 2011-06-30 22:57:05,768 INFO
>>> > org.apache.hadoop.hbase.regionserver.Leases: regionserver50820 closed
>>> > leases
>>> > 2011-06-30 22:57:05,768 INFO
>>> >
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>>> > Closed zookeeper sessionid=0x130ba69074900b4
>>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ZooKeeper: Session:
>>> > 0x130ba69074900b4 closed
>>> > 2011-06-30 22:57:05,781 INFO org.apache.zookeeper.ClientCnxn:
>>> > EventThread shut down
>>> > 2011-06-30 22:57:05,857 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > 2011-06-30 22:57:05,863 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > 2011-06-30 22:57:05,911 INFO
>>> > org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.
>>> > in META
>>> > 2011-06-30 22:57:05,942 INFO
>>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > in region .META.,,1, serverInfo=null
>>> > 2011-06-30 22:57:05,943 INFO
>>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>>> > daughter
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.
>>> > because stopping=false, stopped=true
>>> > 2011-06-30 22:57:05,950 INFO
>>> > org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > in region .META.,,1, serverInfo=null
>>> > 2011-06-30 22:57:05,950 INFO
>>> > org.apache.hadoop.hbase.regionserver.SplitTransaction: Not opening
>>> > daughter
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294.
>>> > because stopping=false, stopped=true
>>> > 2011-06-30 22:57:06,004 INFO
>>> > org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
>>> > updated, and report to master.
>>> >
>>> Parent=CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309422002877.de5cb72653d016804cbd16f4a71470cd.,
>>> > new regions:
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,39999999999999960,1309445753679.e8054c8476b50e7648af747011d0c77e.,
>>> >
>>> CMCC_Detail_ReversePhoneMonth__DateCat_NONE,40277780931201101,1309445753679.64d28c449c062d5ac569f8619a75c294..
>>> > Split took 1mins, 12sec
>>> > 2011-06-30 22:57:06,004 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>>> > Split Thread to finish...
>>> > 2011-06-30 22:57:06,004 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>>> > Large Compaction Thread to finish...
>>> > 2011-06-30 22:57:06,004 DEBUG
>>> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
>>> > Small Compaction Thread to finish...
>>> > 2011-06-30 22:57:06,004 INFO
>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver50820
>>> > exiting
>>> > 2011-06-30 22:57:06,090 INFO
>>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>>> > starting; hbase.shutdown.hook=true;
>>> > fsShutdownHook=Thread[Thread-15,5,main]
>>> > 2011-06-30 22:57:06,090 INFO
>>> > org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown
>>> > hook
>>> > 2011-06-30 22:57:06,090 INFO
>>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs
>>> > shutdown hook thread.
>>> > 2011-06-30 22:57:06,196 INFO
>>> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
>>> > finished.
>>> >
>>> >
>>> > Thanks
>>> > Weihua
>>> >
>>>
>>
>

Mime
View raw message