hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3804) HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
Date Thu, 21 Apr 2011 17:16:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022831#comment-13022831
] 

Jean-Daniel Cryans commented on HBASE-3804:
-------------------------------------------

So how it works: for some reason your region server died. A region server is what carries
and serves the regions (so the data). One thing that's not clear to me is if you are running
in "standalone" mode or "pseudo-distributed" mode. If the former, then yeah I agree that it's
a bit dumb that it locks up like that because you cannot restart the region server _inside_
the master process. If you were using the latter, then well your region server died and you
need to restart it. 

What the master does when a node dies is that it splits the write ahead logs (that's what's
going on in the beginning of your log) and reassigns the regions. In your case, there's no
one to reassign the regions to. The master cannot carry the regions, so it's waiting for a
new region server to come in.

On to why the region server died, there should be a hint in the log. Something on the FATAL
log level. Probably GC pauses.

BTW the book has this to say on single node clusters: "Do not use this configuration for production
nor for evaluating HBase performance."

> HBase gets stuck with "Regions in transition timed out" for .META. and -ROOT-
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-3804
>                 URL: https://issues.apache.org/jira/browse/HBASE-3804
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.90.2
>         Environment: CentOs 5.4 x64 kernel 2.6.18-238.5.1.el5
> Single node cluster
>            Reporter: Laurent Chavet
>            Priority: Blocker
>
> After putting some significant load on the sever (load around of 8 for approximatively
50 minutes) the following starts showing in the logs and then hbase is completely stuck: won't
recover without a restart (I copy a large chunk of the master logs before the problem in case
it helps; the bottom part about ROOT and META repeats forever):
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,228 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter:
Using syncFs -- HDFS-200
> 2011-04-20 06:19:51,228 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012
region=69e1f895ca9d6824b409
> 015a9f9c03a3
> 2011-04-20 06:19:51,539 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Pushed=38 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:51,540 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Splitting hlog 42 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251,
length=67522024
> 2011-04-20 06:19:51,540 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:52,543 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover
attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:53,336 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Pushed=40 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:54,894 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Splitting hlog 43 of 43: hdfs://HQS
> NOSQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805,
length=75765855
> 2011-04-20 06:19:54,906 INFO org.apache.hadoop.hbase.util.FSUtils: Recovering file hdfs://HQSNOSQL01:9000/hbase/.logs/HQ
> SNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:55,108 INFO org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter:
Using syncFs -- HDFS-200
> 2011-04-20 06:19:55,108 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Creating writer path=hdfs://hqsnosq
> l01:9000/hbase/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062
region=a21bf162e9ff2915ff03
> 6f486919c9b5
> 2011-04-20 06:19:55,908 INFO org.apache.hadoop.hbase.util.FSUtils: Finished lease recover
attempt for hdfs://HQSNOSQL01:
> 9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,842 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
Pushed=43 entries from hdfs://HQSNO
> SQL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:56,859 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305143661
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305143661
> 2011-04-20 06:19:56,863 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305152287
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305152287
> 2011-04-20 06:19:56,867 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305160979
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305160979
> 2011-04-20 06:19:56,871 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305170430
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305170430
> 2011-04-20 06:19:56,875 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305178662
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305178662
> 2011-04-20 06:19:56,879 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305188551
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305188551
> 2011-04-20 06:19:56,883 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305196703
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305196703
> 2011-04-20 06:19:56,887 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305205039
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305205039
> 2011-04-20 06:19:56,891 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305213412
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305213412
> 2011-04-20 06:19:56,895 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305220665
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305220665
> 2011-04-20 06:19:56,899 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305231697
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305231697
> 2011-04-20 06:19:56,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305239174
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305239174
> 2011-04-20 06:19:56,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305248013
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305248013
> 2011-04-20 06:19:56,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305256411
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305256411
> 2011-04-20 06:19:56,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305265663
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305265663
> 2011-04-20 06:19:56,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305274667
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305274667
> 2011-04-20 06:19:56,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305283140
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305283140
> 2011-04-20 06:19:56,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305291307
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305291307
> 2011-04-20 06:19:56,941 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305300632
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305300632
> 2011-04-20 06:19:56,945 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305309439
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305309439
> 2011-04-20 06:19:56,949 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305316405
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305316405
> 2011-04-20 06:19:56,953 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305322730
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305322730
> 2011-04-20 06:19:56,957 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305330453
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305330453
> 2011-04-20 06:19:56,961 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305337020
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305337020
> 2011-04-20 06:19:56,965 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305345309
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305345309
> 2011-04-20 06:19:56,969 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305352766
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305352766
> 2011-04-20 06:19:56,972 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305362335
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305362335
> 2011-04-20 06:19:56,976 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305370474
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305370474
> 2011-04-20 06:19:56,980 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305379360
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305379360
> 2011-04-20 06:19:56,984 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305387434
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305387434
> 2011-04-20 06:19:56,988 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305396557
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305396557
> 2011-04-20 06:19:56,992 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305403848
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305403848
> 2011-04-20 06:19:56,996 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305412950
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305412950
> 2011-04-20 06:19:57,000 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305420349
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305420349
> 2011-04-20 06:19:57,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305429358
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305429358
> 2011-04-20 06:19:57,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305437383
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305437383
> 2011-04-20 06:19:57,012 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305446715
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305446715
> 2011-04-20 06:19:57,016 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305460169
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305460169
> 2011-04-20 06:19:57,678 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305467377
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305467377
> 2011-04-20 06:19:57,685 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305476873
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305476873
> 2011-04-20 06:19:57,689 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305484717
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305484717
> 2011-04-20 06:19:57,693 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305492251
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305492251
> 2011-04-20 06:19:57,697 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://HQSNOS
> QL01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146/HQSNOSQL01%3A60020.1303305500805
to hdfs://hqsnosql01:9000/hbase/.o
> ldlogs/HQSNOSQL01%3A60020.1303305500805
> 2011-04-20 06:19:57,701 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Waiting
for split writer threads to
> finish
> 2011-04-20 06:19:57,874 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Split
writers finished
> 2011-04-20 06:19:57,881 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/36d89fc46915d8db56bad45c64a313a3/recovered.edits/0000000000000096979 (wrote
134 edits in 6107ms)
> 2011-04-20 06:19:57,885 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/4499b405bae1047884256f39b7559376/recovered.edits/0000000000000096571 (wrote
30 edits in 1338ms)
> 2011-04-20 06:19:57,889 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/465fb6b0a1f2d5306c7bb87145a32be2/recovered.edits/0000000000000096568 (wrote
22 edits in 909ms)
> 2011-04-20 06:19:57,893 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/4bc3c0bd3a824fb6c7fabbd332ef1009/recovered.edits/0000000000000097151 (wrote
85 edits in 3982ms)
> 2011-04-20 06:19:57,897 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/4bfff6ceb59cc4b6098eea42b49ee9ed/recovered.edits/0000000000000097336 (wrote
117 edits in 4810ms)
> 2011-04-20 06:19:57,901 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/52dee69cbe0890b54aee56bda54577c9/recovered.edits/0000000000000096567 (wrote
63 edits in 4445ms)
> 2011-04-20 06:19:57,905 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/539e4afe8064897395d4dc891605e702/recovered.edits/0000000000000097127 (wrote
53 edits in 3433ms)
> 2011-04-20 06:19:57,909 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/5728cb0bb67c7c3d8701fc0679db97b9/recovered.edits/0000000000000097473 (wrote
134 edits in 6623ms)
> 2011-04-20 06:19:57,913 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/5f92d0c1defc108140df27322257584d/recovered.edits/0000000000000096672 (wrote
77 edits in 7211ms)
> 2011-04-20 06:19:57,917 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/69e1f895ca9d6824b409015a9f9c03a3/recovered.edits/0000000000000098012 (wrote
10 edits in 306ms)
> 2011-04-20 06:19:57,921 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/70746a106ab152f290264d0b2b0f773c/recovered.edits/0000000000000096722 (wrote
117 edits in 7245ms)
> 2011-04-20 06:19:57,925 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/906e3283419ae67052c34e31cc40ff05/recovered.edits/0000000000000097951 (wrote
31 edits in 2389ms)
> 2011-04-20 06:19:57,929 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/94112f821b90271f9d693d5505d5eebf/recovered.edits/0000000000000097692 (wrote
71 edits in 4771ms)
> 2011-04-20 06:19:57,933 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/9f519dfd7370990d017e7f6d187968d3/recovered.edits/0000000000000096697 (wrote
77 edits in 5551ms)
> 2011-04-20 06:19:57,937 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/9f60e52d384d0fd9dfc2a2f3be42ac43/recovered.edits/0000000000000097877 (wrote
24 edits in 840ms)
> 2011-04-20 06:19:58,054 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/a21bf162e9ff2915ff036f486919c9b5/recovered.edits/0000000000000098062 (wrote
8 edits in 1860ms)
> 2011-04-20 06:19:58,061 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/abc26969d71b22de80c7e889cc9a6c12/recovered.edits/0000000000000096576 (wrote
5 edits in 281ms)
> 2011-04-20 06:19:58,065 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/b5d7c2bd8383ea8871890f40b9827dcf/recovered.edits/0000000000000097554 (wrote
125 edits in 5240ms)
> 2011-04-20 06:19:58,073 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/bf06b369cf84e5bccdd4e845636a2af6/recovered.edits/0000000000000096839 (wrote
1 edits in 17ms)
> 2011-04-20 06:19:58,077 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/cf46d330cdaa1120c8e1a5fcf1899b0f/recovered.edits/0000000000000097442 (wrote
1 edits in 850ms)
> 2011-04-20 06:19:58,081 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/d850f84bfdfa28c63934a0be5a41257e/recovered.edits/0000000000000096572 (wrote
53 edits in 2467ms)
> 2011-04-20 06:19:58,085 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/d9b94307130a7096032b44b3aa18d3ea/recovered.edits/0000000000000096795 (wrote
155 edits in 10534ms)
> 2011-04-20 06:19:58,088 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/da6540fdaae565a6d04cd78a145240dd/recovered.edits/0000000000000097580 (wrote
53 edits in 2992ms)
> 2011-04-20 06:19:58,092 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/df02c5d923c466c5b89337fecee920cf/recovered.edits/0000000000000096924 (wrote
1 edits in 25ms)
> 2011-04-20 06:19:58,096 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed
path hdfs://hqsnosql01:9000/h
> base/Property/e90c1c020e3148756a57d977133e31fc/recovered.edits/0000000000000096569 (wrote
96 edits in 3862ms)
> 2011-04-20 06:19:58,097 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog
file splitting completed in 866
> 02 ms for hdfs://hqsnosql01:9000/hbase/.logs/HQSNOSQL01,60020,1303302276146
> 2011-04-20 06:19:58,102 INFO org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting
ROOT region location in ZooKe
> eper
> 2011-04-20 06:19:58,113 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000
Creating (or up
> dating) unassigned node for 70236052 with OFFLINE state
> 2011-04-20 06:19:58,133 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x12f72de7aee0000
Creating (or up
> dating) unassigned node for 1028785192 with OFFLINE state
> 2011-04-20 06:19:58,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE,
>  server=HQSNOSQL01:60000, region=1028785192/.META.
> 2011-04-20 06:20:36,363 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305598113
> 2011-04-20 06:20:36,364 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305598133
> 2011-04-20 06:20:36,364 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305598113
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:06,374 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305636364
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-, 
> ,0.70236052 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305666374
> 2011-04-20 06:21:36,385 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305666374
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:06,393 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305696385
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305726394
> 2011-04-20 06:22:36,402 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305726393
> 2011-04-20 06:22:36,402 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305726394
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:06,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305756402
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519
> 2 state=OFFLINE, ts=1303305786412
> 2011-04-20 06:23:36,420 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=-ROOT-,,0.70236052
> state=OFFLINE, ts=1303305786412
> 2011-04-20 06:24:06,429 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  .META.,
> ,1.1028785192 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning .META.,,1.1028785192 to a random server
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  -ROOT-,
> ,0.70236052 state=OFFLINE, ts=1303305816420
> 2011-04-20 06:24:06,430 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been OFFLINE for too long, rea
> ssigning -ROOT-,,0.70236052 to a random server
> 2011-04-20 06:24:06,430 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing
OFFLINE; was=.META.,,1.102878519

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message