hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4748) Race between creating recovered edits for META and master assigning ROOT and META.
Date Thu, 10 Nov 2011 05:31:52 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147498#comment-13147498 ] 

ramkrishna.s.vasudevan commented on HBASE-4748:
-----------------------------------------------

Stack trace as requested by Ted
Master is restarting and the RS goes down and comes back (holds META and root)

RS logs
{code}
2011-11-10 17:04:31,536 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: -ROOT-,,0.70236052
2011-11-10 17:04:31,747 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 70236052/-ROOT- from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-11-10 17:04:31,789 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 70236052 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-11-10 17:04:31,818 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => '-ROOT-,,0', TableName => '-ROOT-', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}
2011-11-10 17:04:31,856 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
2011-11-10 17:04:31,994 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated -ROOT-,,0.70236052
2011-11-10 17:04:32,933 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://linux76:9000/hbase/-ROOT-/70236052/info/b015599976a2484bb85564ee399976cb, isReference=false, isBulkLoadResult=false, seqid=2, majorCompaction=false
2011-11-10 17:04:33,371 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://linux76:9000/hbase/-ROOT-/70236052/recovered.edits/0000000000000000004; minSequenceid=2; path=hdfs://linux76:9000/hbase/-ROOT-/70236052/recovered.edits/0000000000000000004
2011-11-10 17:04:33,481 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-11-10 17:04:33,505 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-11-10 17:04:33,505 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Applied 2, skipped 0, firstSequenceidInLog=4, maxSequenceidInLog=4, path=hdfs://linux76:9000/hbase/-ROOT-/70236052/recovered.edits/0000000000000000004
2011-11-10 17:04:33,517 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for -ROOT-,,0.70236052, current region memstore size 352.0; wal is null, using passed sequenceid=4
2011-11-10 17:04:33,538 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing waiting for rwcc
2011-11-10 17:04:33,538 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, commencing flushing stores
2011-11-10 17:04:33,777 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: SplitLogWorker linux76,60020,1320924852399 starting
2011-11-10 17:04:33,794 DEBUG org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2011-11-10 17:04:33,794 DEBUG org.apache.hadoop.hbase.util.BloomFilterFactory: Bloom filter is turned off for the column family
2011-11-10 17:04:33,800 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom filter type for hdfs://linux76:9000/hbase/-ROOT-/70236052/.tmp/a2c290a9cb264d99a128a18943cf0de9: CompoundBloomFilterWriter
2011-11-10 17:04:33,803 DEBUG org.apache.hadoop.hbase.io.hfile.HFileBlockIndex: Wrote a 1-level index with root level at pos 200, 1 root-level entries, 1 total entries, 44.0 on-disk size, 44.0 total uncompressed size.
2011-11-10 17:04:33,803 INFO org.apache.hadoop.hbase.io.hfile.HFileBlockIndex: Wrote a single-level meta index with 0 entries, 0 bytes
2011-11-10 17:04:33,873 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and NO DeleteFamily was added to HFile (hdfs://linux76:9000/hbase/-ROOT-/70236052/.tmp/a2c290a9cb264d99a128a18943cf0de9) 
2011-11-10 17:04:33,873 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=4, memsize=352.0, into tmp file hdfs://linux76:9000/hbase/-ROOT-/70236052/.tmp/a2c290a9cb264d99a128a18943cf0de9
2011-11-10 17:04:33,891 DEBUG org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://linux76:9000/hbase/-ROOT-/70236052/.tmp/a2c290a9cb264d99a128a18943cf0de9 to hdfs://linux76:9000/hbase/-ROOT-/70236052/info/a2c290a9cb264d99a128a18943cf0de9
2011-11-10 17:04:33,910 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://linux76:9000/hbase/-ROOT-/70236052/info/a2c290a9cb264d99a128a18943cf0de9, entries=2, sequenceid=4, filesize=719.0
2011-11-10 17:04:33,911 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~352.0 for region -ROOT-,,0.70236052 in 394ms, sequenceid=4, compaction requested=false; wal=null
2011-11-10 17:04:33,929 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Deleted recovered.edits file=hdfs://linux76:9000/hbase/-ROOT-/70236052/recovered.edits/0000000000000000004
2011-11-10 17:04:34,616 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined -ROOT-,,0.70236052; next sequenceid=5
2011-11-10 17:04:34,616 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-11-10 17:04:34,621 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-11-10 17:04:34,627 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=-ROOT-,,0.70236052, daughter=false
2011-11-10 17:04:34,627 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: addToOnlineRegions is doneREGION => {NAME => '-ROOT-,,0', TableName => '-ROOT-', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}
2011-11-10 17:04:34,630 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Setting ROOT region location in ZooKeeper as linux76,60020,1320924852399
2011-11-10 17:04:34,642 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Done with post open deploy taks for region=-ROOT-,,0.70236052, daughter=false
2011-11-10 17:04:34,643 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 70236052/-ROOT- from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-11-10 17:04:34,652 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 70236052 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-11-10 17:04:34,653 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: region transitioned to opened in zookeeper: REGION => {NAME => '-ROOT-,,0', TableName => '-ROOT-', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}, server: linux76,60020,1320924852399
2011-11-10 17:04:34,653 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened -ROOT-,,0.70236052 on server:linux76,60020,1320924852399
2011-11-10 17:04:35,863 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: .META.,,1.1028785192
2011-11-10 17:04:35,881 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 1028785192/.META. from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-11-10 17:04:35,891 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 1028785192 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2011-11-10 17:04:35,891 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2011-11-10 17:04:35,892 INFO org.apache.hadoop.hbase.regionserver.HRegion: Setting up tabledescriptor config now ...
2011-11-10 17:04:35,893 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated .META.,,1.1028785192
2011-11-10 17:04:35,919 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined .META.,,1.1028785192; next sequenceid=1
2011-11-10 17:04:35,919 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-11-10 17:04:36,005 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING
2011-11-10 17:04:36,045 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks for region=.META.,,1.1028785192, daughter=false
2011-11-10 17:04:36,045 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: addToOnlineRegions is doneREGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2011-11-10 17:04:36,199 DEBUG org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or departed
2011-11-10 17:04:36,246 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 acquired task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350
2011-11-10 17:04:36,378 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924693350, length=294
2011-11-10 17:04:36,381 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924693350
2011-11-10 17:04:37,250 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@964823; serverName=linux76,60020,1320924852399
2011-11-10 17:04:37,263 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@964823; serverName=linux76,60020,1320924852399
2011-11-10 17:04:37,394 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924693350
2011-11-10 17:04:37,453 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: new createWriter -- HADOOP-6840 -- not available
2011-11-10 17:04:37,464 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350/-ROOT-/70236052/recovered.edits/0000000000000000004, syncFs=true, hflush=false
2011-11-10 17:04:37,465 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350/-ROOT-/70236052/recovered.edits/0000000000000000004 region=70236052
2011-11-10 17:04:37,482 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: HDFS pipeline error detected. Found 1 replicas but expecting no less than 2 replicas.  Requesting close of hlog.
2011-11-10 17:04:37,483 DEBUG org.apache.hadoop.hbase.regionserver.LogRoller: HLog roll requested
2011-11-10 17:04:37,488 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: new createWriter -- HADOOP-6840 -- not available
2011-11-10 17:04:37,494 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0246 has been closed.
2011-11-10 17:04:37,494 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row .META.,,1.1028785192 with server=linux76,60020,1320924852399
2011-11-10 17:04:37,494 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Done with post open deploy taks for region=.META.,,1.1028785192, daughter=false
2011-11-10 17:04:37,499 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-11-10 17:04:37,504 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Path=hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924852399/linux76%2C60020%2C1320924852399.1320924877487, syncFs=true, hflush=false
2011-11-10 17:04:37,541 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x1335f3da72b0245 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2011-11-10 17:04:37,541 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: region transitioned to opened in zookeeper: REGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}, server: linux76,60020,1320924852399
2011-11-10 17:04:37,542 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opened .META.,,1.1028785192 on server:linux76,60020,1320924852399
2011-11-10 17:04:37,577 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Current cached META location, null, is not valid, resetting
2011-11-10 17:04:37,583 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350/-ROOT-/70236052/recovered.edits/0000000000000000004
2011-11-10 17:04:37,583 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: processed 1 edits across 1 regions threw away edits for 0 regions log file = hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924693350 is corrupted = false
2011-11-10 17:04:37,587 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: Roll /hbase/.logs/linux76,60020,1320924852399/linux76%2C60020%2C1320924852399.1320924853463, entries=1, filesize=294.  for /hbase/.logs/linux76,60020,1320924852399/linux76%2C60020%2C1320924852399.1320924877487
2011-11-10 17:04:37,598 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350 to final state done
2011-11-10 17:04:37,599 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 done with task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350 in 1338ms
2011-11-10 17:04:37,616 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 acquired task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044
2011-11-10 17:04:37,620 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924804044, length=352
2011-11-10 17:04:37,620 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924804044
2011-11-10 17:04:38,539 DEBUG org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or departed
2011-11-10 17:04:38,622 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924804044
2011-11-10 17:04:38,691 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: new createWriter -- HADOOP-6840 -- not available
2011-11-10 17:04:38,697 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000007, syncFs=true, hflush=false
2011-11-10 17:04:38,697 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000007 region=dcbe1ee9681da1111a8043e2d8515dc4
2011-11-10 17:04:38,748 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000007
2011-11-10 17:04:38,748 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: processed 2 edits across 1 regions threw away edits for 0 regions log file = hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924804044 is corrupted = false
2011-11-10 17:04:38,805 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044 to final state done
2011-11-10 17:04:38,805 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 done with task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044 in 1189ms
2011-11-10 17:04:38,833 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 acquired task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282
2011-11-10 17:04:38,837 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806282, length=484
2011-11-10 17:04:38,837 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806282
2011-11-10 17:04:39,842 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806282
2011-11-10 17:04:39,880 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: new createWriter -- HADOOP-6840 -- not available
2011-11-10 17:04:39,901 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000009, syncFs=true, hflush=false
2011-11-10 17:04:39,901 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000009 region=dcbe1ee9681da1111a8043e2d8515dc4
2011-11-10 17:04:39,930 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000009
2011-11-10 17:04:39,930 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: processed 3 edits across 1 regions threw away edits for 0 regions log file = hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806282 is corrupted = false
2011-11-10 17:04:39,942 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282 to final state done
2011-11-10 17:04:39,942 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 done with task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282 in 1109ms
2011-11-10 17:04:39,964 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 acquired task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535
2011-11-10 17:04:39,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806535, length=0
2011-11-10 17:04:39,969 WARN org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806535 might be still open, length is 0
2011-11-10 17:04:39,971 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806535
2011-11-10 17:04:40,983 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806535
2011-11-10 17:04:41,140 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: new createWriter -- HADOOP-6840 -- not available
2011-11-10 17:04:41,162 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000012, syncFs=true, hflush=false
2011-11-10 17:04:41,162 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000012 region=dcbe1ee9681da1111a8043e2d8515dc4
2011-11-10 17:04:41,288 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000012
2011-11-10 17:04:41,288 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: processed 200 edits across 1 regions threw away edits for 0 regions log file = hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806535 is corrupted = false
2011-11-10 17:04:41,295 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535 to final state done
2011-11-10 17:04:41,295 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 done with task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535 in 1330ms
2011-11-10 17:04:41,315 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 acquired task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840
2011-11-10 17:04:41,319 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog: hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924802840, length=380
2011-11-10 17:04:41,319 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Recovering file hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924802840
2011-11-10 17:04:42,323 INFO org.apache.hadoop.hbase.util.FSHDFSUtils: Finished lease recover attempt for hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924802840
2011-11-10 17:04:42,359 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: new createWriter -- HADOOP-6840 -- not available
2011-11-10 17:04:42,431 DEBUG org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840/.META./1028785192/recovered.edits/0000000000000000006, syncFs=true, hflush=false
2011-11-10 17:04:42,432 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840/.META./1028785192/recovered.edits/0000000000000000006 region=1028785192
2011-11-10 17:04:42,582 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840/.META./1028785192/recovered.edits/0000000000000000006
2011-11-10 17:04:42,582 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: processed 1 edits across 1 regions threw away edits for 0 regions log file = hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924802840 is corrupted = false
2011-11-10 17:04:42,597 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully transitioned task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840 to final state done
2011-11-10 17:04:42,597 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker linux76,60020,1320924852399 done with task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840 in 1281ms

{code}
Master logs
{code}
2011-11-10 17:03:55,284 DEBUG org.apache.hadoop.hbase.master.HMaster: Started service threads
2011-11-10 17:03:58,288 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-11-10 17:04:01,292 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-11-10 17:04:04,296 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-11-10 17:04:07,300 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-11-10 17:04:10,313 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) to checkin
2011-11-10 17:04:13,124 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=linux76,60020,1320924852399
2011-11-10 17:04:13,317 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting on regionserver(s) count to settle; currently=1
2011-11-10 17:04:16,321 INFO org.apache.hadoop.hbase.master.HMaster: Registering server found up in zk: linux76,60020,1320924692314
2011-11-10 17:04:16,321 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=linux76,60020,1320924692314
2011-11-10 17:04:22,495 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314 belongs to an existing region server
2011-11-10 17:04:22,497 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924852399 belongs to an existing region server
2011-11-10 17:04:22,497 INFO org.apache.hadoop.hbase.master.MasterFileSystem: No logs to split
2011-11-10 17:04:31,356 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of -ROOT-,,0 at address=linux76,60020,1320924692314; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0
2011-11-10 17:04:31,358 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region location in ZooKeeper
2011-11-10 17:04:31,511 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Creating (or updating) unassigned node for 70236052 with OFFLINE state
2011-11-10 17:04:31,530 INFO org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for -ROOT-,,0.70236052 so generated a random one; hri=-ROOT-,,0.70236052, src=, dest=linux76,60020,1320924852399; 2 (online=2, exclude=0) available servers
2011-11-10 17:04:31,530 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning region -ROOT-,,0.70236052 to linux76,60020,1320924852399
2011-11-10 17:04:31,530 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to linux76,60020,1320924852399
2011-11-10 17:04:31,786 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux76,60020,1320924852399, region=70236052/-ROOT-
2011-11-10 17:04:33,505 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux76,60020,1320924852399, region=70236052/-ROOT-
2011-11-10 17:04:34,008 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [linux76,60020,1320924692314]
2011-11-10 17:04:34,315 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: based on ZK, current region=-ROOT-,,0.70236052 is on server=linux76,60020,1320924852399 server being checked=: linux76,60020,1320924692314
2011-11-10 17:04:34,317 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: based on AM, current region=.META.,,1.1028785192 is on server=null server being checked: linux76,60020,1320924692314
2011-11-10 17:04:34,558 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=linux76,60020,1320924692314 to dead servers, submitted shutdown handler to be executed, root=false, meta=false
2011-11-10 17:04:34,623 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux76,60020,1320924852399, region=70236052/-ROOT-
2011-11-10 17:04:34,654 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=linux76,60020,1320924852399, region=70236052/-ROOT-
2011-11-10 17:04:34,658 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for -ROOT-,,0.70236052 from linux76,60020,1320924852399; deleting unassigned node
2011-11-10 17:04:34,658 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Deleting existing unassigned node for 70236052 that is in expected state RS_ZK_REGION_OPENED
2011-11-10 17:04:34,670 WARN org.apache.hadoop.hbase.master.AssignmentManager: Node deleted but still in RIT: -ROOT-,,0.70236052 state=OPEN, ts=1320924874644, server=linux76,60020,1320924852399
2011-11-10 17:04:34,670 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Successfully deleted unassigned node for region 70236052 in expected state RS_ZK_REGION_OPENED
2011-11-10 17:04:34,679 INFO org.apache.hadoop.hbase.master.HMaster: -ROOT- assigned=1, rit=false, location=linux76,60020,1320924852399
2011-11-10 17:04:34,729 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:34,736 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:34,761 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The master has opened the region -ROOT-,,0.70236052 that was online on linux76,60020,1320924852399
2011-11-10 17:04:34,793 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:34,813 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=linux76,60020,1320924159959; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2011-11-10 17:04:34,813 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: new .META. server: linux76,60020,1320924159959 isn't valid. Cached .META. server: null
2011-11-10 17:04:35,317 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:35,320 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:35,324 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:35,330 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=linux76,60020,1320924159959; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2011-11-10 17:04:35,330 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: new .META. server: linux76,60020,1320924159959 isn't valid. Cached .META. server: null
2011-11-10 17:04:35,833 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:35,834 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:35,839 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:35,846 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=linux76,60020,1320924159959; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2011-11-10 17:04:35,846 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: new .META. server: linux76,60020,1320924159959 isn't valid. Cached .META. server: null
2011-11-10 17:04:35,850 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Creating (or updating) unassigned node for 1028785192 with OFFLINE state
2011-11-10 17:04:35,859 INFO org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=linux76,60020,1320924852399; 1 (online=1, exclude=0) available servers
2011-11-10 17:04:35,859 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning region .META.,,1.1028785192 to linux76,60020,1320924852399
2011-11-10 17:04:35,859 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to linux76,60020,1320924852399
2011-11-10 17:04:35,865 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=linux76,60000,1320924832712, region=1028785192/.META.
2011-11-10 17:04:35,870 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:35,871 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:35,875 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:35,880 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=linux76,60020,1320924159959; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2011-11-10 17:04:35,880 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: new .META. server: linux76,60020,1320924159959 isn't valid. Cached .META. server: null
2011-11-10 17:04:35,890 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux76,60020,1320924852399, region=1028785192/.META.
2011-11-10 17:04:36,006 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:36,007 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=linux76,60020,1320924852399, region=1028785192/.META.
2011-11-10 17:04:36,009 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:36,015 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:36,018 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=linux76,60020,1320924159959; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2011-11-10 17:04:36,018 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: new .META. server: linux76,60020,1320924159959 isn't valid. Cached .META. server: null
2011-11-10 17:04:36,021 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Retrying
org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException: Timed out (100ms)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:449)
	at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:413)
	at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:563)
	at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:485)
	at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:322)
	at java.lang.Thread.run(Thread.java:619)
2011-11-10 17:04:36,027 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:36,029 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:36,040 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:36,045 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of .META.,,1 at address=linux76,60020,1320924159959; org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: .META.,,1
2011-11-10 17:04:36,045 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: new .META. server: linux76,60020,1320924159959 isn't valid. Cached .META. server: null
2011-11-10 17:04:36,114 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs for linux76,60020,1320924692314
2011-11-10 17:04:36,139 DEBUG org.apache.hadoop.hbase.master.MasterFileSystem: Renamed region directory: hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting
2011-11-10 17:04:36,139 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog worker linux76,60020,1320924692314
2011-11-10 17:04:36,149 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:36,151 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:36,155 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:36,161 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: set new cached META location: linux76,60020,1320924159959
2011-11-10 17:04:36,181 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: Scheduling batch of logs to split
2011-11-10 17:04:36,183 INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting logs in [hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting]
2011-11-10 17:04:36,207 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350
2011-11-10 17:04:36,221 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869
2011-11-10 17:04:36,223 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840
2011-11-10 17:04:36,224 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044
2011-11-10 17:04:36,225 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282
2011-11-10 17:04:36,226 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at znode /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535
2011-11-10 17:04:36,227 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350 ver = 0
2011-11-10 17:04:36,229 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869 ver = 0
2011-11-10 17:04:36,230 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840 ver = 0
2011-11-10 17:04:36,231 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044 ver = 0
2011-11-10 17:04:36,231 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282 ver = 0
2011-11-10 17:04:36,232 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535 ver = 0
2011-11-10 17:04:36,241 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350 acquired by linux76,60020,1320924852399
2011-11-10 17:04:36,713 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 6 unassigned = 5
2011-11-10 17:04:37,529 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=linux76,60020,1320924852399, region=1028785192/.META.
2011-11-10 17:04:37,531 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for .META.,,1.1028785192 from linux76,60020,1320924852399; deleting unassigned node
2011-11-10 17:04:37,532 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED
2011-11-10 17:04:37,549 WARN org.apache.hadoop.hbase.master.AssignmentManager: Node deleted but still in RIT: .META.,,1.1028785192 state=OPEN, ts=1320924877527, server=linux76,60020,1320924852399
2011-11-10 17:04:37,549 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Current cached META location, linux76,60020,1320924159959, is not valid, resetting
2011-11-10 17:04:37,578 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED
2011-11-10 17:04:37,578 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The master has opened the region .META.,,1.1028785192 that was online on linux76,60020,1320924852399
2011-11-10 17:04:37,581 INFO org.apache.hadoop.hbase.master.HMaster: .META. assigned=2, rit=false, location=linux76,60020,1320924159959
2011-11-10 17:04:37,600 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350 entered state done linux76,60020,1320924852399
2011-11-10 17:04:37,624 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:37,631 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over -ROOT- starting at key ''
2011-11-10 17:04:37,631 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2011-11-10 17:04:37,634 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:37,666 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at REGION => {NAME => '-ROOT-,,0', TableName => '-ROOT-', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,}
2011-11-10 17:04:37,667 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:37,667 INFO org.apache.hadoop.hbase.catalog.MetaMigrationRemovingHTD: Meta version=0; migrated=true
2011-11-10 17:04:37,667 INFO org.apache.hadoop.hbase.catalog.MetaMigrationRemovingHTD: ROOT/Meta already up-to date with new HRI.
2011-11-10 17:04:37,672 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:37,698 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is linux76:60020
2011-11-10 17:04:37,699 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over .META. starting at key ''
2011-11-10 17:04:37,699 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2011-11-10 17:04:37,705 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at REGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2011-11-10 17:04:37,706 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:37,707 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning userregions
2011-11-10 17:04:37,707 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x1335f3da72b0243 Deleting any existing unassigned nodes
2011-11-10 17:04:37,711 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over .META. starting at key ''
2011-11-10 17:04:37,711 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2011-11-10 17:04:37,715 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 6 unassigned = 5
2011-11-10 17:04:37,721 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at REGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2011-11-10 17:04:37,721 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:38,135 INFO org.apache.hadoop.hbase.master.HMaster: Master has completed initialization
2011-11-10 17:04:38,180 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over .META. starting at key ''
2011-11-10 17:04:38,182 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2011-11-10 17:04:38,194 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at REGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2011-11-10 17:04:38,194 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:38,194 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s) and gc'd 0 unreferenced parent region(s)
2011-11-10 17:04:38,211 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:  moved hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350/-ROOT-/70236052/recovered.edits/0000000000000000004 => hdfs://linux76:9000/hbase/-ROOT-/70236052/recovered.edits/0000000000000000004
2011-11-10 17:04:38,513 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924693350 to hdfs://linux76:9000/hbase/.oldlogs/linux76%2C60020%2C1320924692314.1320924693350
2011-11-10 17:04:38,534 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350
2011-11-10 17:04:38,540 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924693350
2011-11-10 17:04:38,540 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044 acquired by linux76,60020,1320924852399
2011-11-10 17:04:38,719 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 5 unassigned = 4
2011-11-10 17:04:38,758 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044 entered state done linux76,60020,1320924852399
2011-11-10 17:04:38,841 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:  moved hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000007 => hdfs://linux76:9000/hbase/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000007
2011-11-10 17:04:38,880 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924804044 to hdfs://linux76:9000/hbase/.oldlogs/linux76%2C60020%2C1320924692314.1320924804044
2011-11-10 17:04:38,890 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044
2011-11-10 17:04:38,916 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924804044
2011-11-10 17:04:38,916 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282 acquired by linux76,60020,1320924852399
2011-11-10 17:04:39,722 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 4 unassigned = 3
2011-11-10 17:04:39,944 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282 entered state done linux76,60020,1320924852399
2011-11-10 17:04:40,014 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:  moved hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000009 => hdfs://linux76:9000/hbase/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000009
2011-11-10 17:04:40,060 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806282 to hdfs://linux76:9000/hbase/.oldlogs/linux76%2C60020%2C1320924692314.1320924806282
2011-11-10 17:04:40,063 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282
2011-11-10 17:04:40,076 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806282
2011-11-10 17:04:40,077 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535 acquired by linux76,60020,1320924852399
2011-11-10 17:04:40,727 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 3 unassigned = 2
2011-11-10 17:04:41,296 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535 entered state done linux76,60020,1320924852399
2011-11-10 17:04:41,348 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:  moved hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000012 => hdfs://linux76:9000/hbase/t1/dcbe1ee9681da1111a8043e2d8515dc4/recovered.edits/0000000000000000012
2011-11-10 17:04:41,370 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924806535 to hdfs://linux76:9000/hbase/.oldlogs/linux76%2C60020%2C1320924692314.1320924806535
2011-11-10 17:04:41,378 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535
2011-11-10 17:04:41,386 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924806535
2011-11-10 17:04:41,386 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840 acquired by linux76,60020,1320924852399
2011-11-10 17:04:41,731 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 2 unassigned = 1
2011-11-10 17:04:42,598 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840 entered state done linux76,60020,1320924852399
2011-11-10 17:04:42,680 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:  moved hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840/.META./1028785192/recovered.edits/0000000000000000006 => hdfs://linux76:9000/hbase/.META./1028785192/recovered.edits/0000000000000000006
2011-11-10 17:04:42,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924802840 to hdfs://linux76:9000/hbase/.oldlogs/linux76%2C60020%2C1320924692314.1320924802840
2011-11-10 17:04:42,705 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840
2011-11-10 17:04:42,713 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924802840
2011-11-10 17:04:42,714 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869 acquired by linux76,60020,1320924852399
2011-11-10 17:04:42,735 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 0
2011-11-10 17:04:43,725 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869 entered state done linux76,60020,1320924852399
2011-11-10 17:04:43,739 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 0
2011-11-10 17:04:43,764 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:  moved hdfs://linux76:9000/hbase/splitlog/linux76,60020,1320924852399_hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869/.META./1028785192/recovered.edits/0000000000000000005 => hdfs://linux76:9000/hbase/.META./1028785192/recovered.edits/0000000000000000005
2011-11-10 17:04:43,791 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed log hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting/linux76%2C60020%2C1320924692314.1320924792869 to hdfs://linux76:9000/hbase/.oldlogs/linux76%2C60020%2C1320924692314.1320924792869
2011-11-10 17:04:43,799 INFO org.apache.hadoop.hbase.master.SplitLogManager: Done splitting /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869
2011-11-10 17:04:43,799 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: re-listing hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting
2011-11-10 17:04:43,808 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/hdfs%3A%2F%2Flinux76%3A9000%2Fhbase%2F.logs%2Flinux76%2C60020%2C1320924692314-splitting%2Flinux76%252C60020%252C1320924692314.1320924792869
2011-11-10 17:04:43,810 INFO org.apache.hadoop.hbase.master.SplitLogManager: finished splitting (more than or equal to) 1851 bytes in 6 log files in [hdfs://linux76:9000/hbase/.logs/linux76,60020,1320924692314-splitting] in 7627ms
2011-11-10 17:04:46,215 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:46,218 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Lookedup root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@fe8c4; serverName=linux76,60020,1320924852399
2011-11-10 17:04:46,221 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:46,223 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: set new cached META location: linux76,60020,1320924852399
2011-11-10 17:04:46,224 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner over .META. starting at key ''
2011-11-10 17:04:46,224 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal scanner to startKey at ''
2011-11-10 17:04:46,231 DEBUG org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with scanning at REGION => {NAME => '.META.,,1', TableName => '.META.', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
2011-11-10 17:04:46,231 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x1335f3da72b0244 has been closed.
2011-11-10 17:04:46,231 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Reassigning 0 region(s) that linux76,60020,1320924692314 was carrying (skipping 0 regions(s) that are already in transition)
2011-11-10 17:04:46,232 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished processing of shutdown of linux76,60020,1320924692314
{code}
                
> Race between creating recovered edits for META and master assigning ROOT and META.
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-4748
>                 URL: https://issues.apache.org/jira/browse/HBASE-4748
>             Project: HBase
>          Issue Type: Bug
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>
> 1. Start a cluster.
> 2. Alter a table
> 3. Restart the master using ./hbase-daemon.sh restart master
> 4. Kill the RS after master restarts.
> 5. Start RS again.
> 6. No table operations can be performed on the table that was altered but admin.listTables() is able to list the altered table.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message