hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chenjiajun (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8912) [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE
Date Thu, 28 Nov 2013 08:55:38 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834617#comment-13834617
] 

chenjiajun commented on HBASE-8912:
-----------------------------------

2013-11-27 18:26:18,102 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort:
loaded coprocessors are: []
2013-11-27 18:26:18,102 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7.
state=PENDING_O
PEN, ts=1385547978102, server=d199.uuc.com,60020,1385047501649 .. Cannot transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state : H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7.
state=PENDING_OPEN, ts=1385547978102, server=d199.uuc.com,60020,1385047501649 .. Cannot transit
it to OFFLINE.
        at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1890)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1690)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398)
        at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393)
        at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
2013-11-27 18:26:18,104 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2013-11-27 18:26:18,104 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2013-11-27 18:26:18,104 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000:
exiting
......

RS 's log:
2013-11-27 18:24:33,375 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family
Bloom filter type for hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d:
CompoundBloomFilterWriter
2013-11-27 18:24:33,385 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom
and NO DeleteFamily was added to HFile (hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d)
2013-11-27 18:24:33,388 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted
file at hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/.tmp/832ec249071c45b3934a186046ca429d
to hdfs://master.uc.uuc.com:9000/hbase/H/18c9cb11b3e673dec07038f166fb3ef7/page/832ec249071c45b3934a186046ca429d
2013-11-27 18:24:33,421 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction
of 3 file(s) in page of H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7.
into 832ec249071c45b3934a186046ca429d, size=85.7k; total size for store is 533.2m
2013-11-27 18:24:33,422 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
completed compaction: regionName=H,http://istock.jrj.com.cn/article,002024,6567377.html,1385541132079.18c9cb11b3e673dec07038f166fb3ef7.,
storeName=page, fileCount=3, fileSize=91.8k, priority=3, time=1230511587222436; duration=0sec
2013-11-27 18:24:37,189 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher
requesting flush for region DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5.
after a delay of 5137
2013-11-27 18:24:43,658 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
getDefaultReplication
2013-11-27 18:24:43,658 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
getDefaultBlockSize
2013-11-27 18:24:43,681 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family
Bloom filter type for hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364:
CompoundBloomFilterWriter
2013-11-27 18:24:43,691 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom
and NO DeleteFamily was added to HFile (hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364)
2013-11-27 18:24:43,691 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=26736748922,
memsize=832.0, into tmp file hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/dd7c3908111d4fcc8945b2b6d5291364
2013-11-27 18:24:43,699 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/data/dd7c3908111d4fcc8945b2b6d5291364,
entries=4, sequenceid=26736748922, filesize=1000.0
2013-11-27 18:24:43,700 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore
flush of ~832.0/832, currentsize=0.0/0 for region DocSortByHour,\x00\x00\x011\xAE\xCB\xD8
\x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5.
in 44ms, sequenceid=26736748922, compaction requested=true
2013-11-27 18:24:43,701 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction
on data in region DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5.
2013-11-27 18:24:43,701 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction
of 3 file(s) in data of DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5.
into tmpdir=hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp,
seqid=26736748922, totalSize=47.9m
2013-11-27 18:24:43,755 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
getDefaultReplication
2013-11-27 18:24:43,756 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
getDefaultBlockSize
2013-11-27 18:24:43,757 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family
Bloom filter type for hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/c2047a8a560645af936bcbcfe2625e5e:
CompoundBloomFilterWriter
2013-11-27 18:24:47,354 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom
and DeleteFamily was added to HFile (hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/c2047a8a560645af936bcbcfe2625e5e)
2013-11-27 18:24:47,359 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded
Delete Family Bloom (CompoundBloomFilter) metadata for c2047a8a560645af936bcbcfe2625e5e
2013-11-27 18:24:47,359 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted
file at hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/.tmp/c2047a8a560645af936bcbcfe2625e5e
to hdfs://master.uc.uuc.com:9000/hbase/DocSortByHour/c5e2ce4602fb62714c72cd8da3b50bd5/data/c2047a8a560645af936bcbcfe2625e5e
2013-11-27 18:24:47,364 INFO org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded
Delete Family Bloom (CompoundBloomFilter) metadata for c2047a8a560645af936bcbcfe2625e5e
2013-11-27 18:24:47,404 INFO org.apache.hadoop.hbase.regionserver.Store: Completed compaction
of 3 file(s) in data of DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5.
into c2047a8a560645af936bcbcfe2625e5e, size=47.9m; total size for store is 223.8m
2013-11-27 18:24:47,404 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
completed compaction: regionName=DocSortByHour,\x00\x00\x011\xAE\xCB\xD8 \x00\x00,B\x00\x00\x05\x84\x00\x01\xE6\xAF\x00\x02\x09\x0D\x00\x00\x00\x00\x06\xB5c\xB3,1314093149893.c5e2ce4602fb62714c72cd8da3b50bd5.,
storeName=data, fileCount=3, fileSize=47.9m, priority=3, time=1230521954697044; duration=3sec
2013-11-27 18:27:07,191 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher
requesting flush for region h,!\xACU\xD2,1375542268855.ae4d6fae898f79e9fa019b78c975dd95. after
a delay of 10647
2013-11-27 18:27:17,190 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020.periodicFlusher
requesting flush for region h,!\xACU\xD2,1375542268855.ae4d6fae898f79e9fa019b78c975dd95. after
a delay of 19328
2013-11-27 18:27:17,841 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
getDefaultReplication
2013-11-27 18:27:17,841 INFO org.apache.hadoop.hbase.util.FSUtils: FileSystem doesn't support
getDefaultBlockSize
2013-11-27 18:27:17,851 INFO org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family
Bloom filter type for hdfs://master.uc.uuc.com:9000/hbase/h/ae4d6fae898f79e9fa019b78c975dd95/.tmp/135b6223aa9a41c1bf3511c7a389baea:
CompoundBloomFilterWriter
2013-11-27 18:27:17,863 INFO org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom
and NO DeleteFamily was added to HFile (hdfs://master.uc.uuc.com:9000/hbase/h/ae4d6fae898f79e9fa019b78c975dd95/.tmp/135b6223aa9a41c1bf3511c7a389baea)

------
and the Master worked well If I used version 0.94.3 ship with RS used version 0.94.13 .

> [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to OFFLINE
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-8912
>                 URL: https://issues.apache.org/jira/browse/HBASE-8912
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Enis Soztutar
>             Fix For: 0.94.15
>
>         Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html
>
>
> AM throws this exception which subsequently causes the master to abort: 
> {code}
> java.lang.IllegalStateException: Unexpected state : testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b.
state=PENDING_OPEN, ts=1372891751912, server=hemera.apache.org,39064,1372891746132 .. Cannot
transit it to OFFLINE.
> 	at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
> 	at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
> 	at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
> 	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> 	at java.lang.Thread.run(Thread.java:662)
> {code}
> This exception trace is from the failing test TestMetaReaderEditor which is failing pretty
frequently, but looking at the test code, I think this is not a test-only issue, but affects
the main code path. 
> https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message