hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Marc Spaggiari <jean-m...@spaggiari.org>
Subject AssignmentManager looping?
Date Thu, 01 Aug 2013 12:22:54 GMT
My master keep logging that:

2013-07-31 21:52:59,201 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,201 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,339 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,339 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,461 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,461 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:52:59,636 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:52:59,636 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,074 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,074 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,261 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,261 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 21:53:00,417 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 21:53:00,417 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split

hbase@node3:~/hbase-0.94.3$ cat logs/hbase-hbase-master-node3.log* | grep
"Region 270a9c371fcbe9cd9a04986e0b77d16b not found " | wc
   5042   65546  927728


Then crashed.
2013-07-31 22:22:46,072 FATAL org.apache.hadoop.hbase.master.HMaster:
Master server abort: loaded coprocessors are: []
2013-07-31 22:22:46,073 FATAL org.apache.hadoop.hbase.master.HMaster:
Unexpected state : work_proposed,\x02\xE8\x92'\x00\x00\x00\x00
http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 .. Cannot
transit it to OFFLINE.
java.lang.IllegalStateException: Unexpected state :
work_proposed,\x02\xE8\x92'\x00\x00\x00\x00
http://video.inportnews.ca/search/all/source/sun-news-network/harry-potter-in-translation/68463493001/page/1526,1375307272709.d95bb27cc026511c2a8c8ad155e79bf6.
state=OPENING, ts=1375323766008, server=node7,60020,1375319044055 .. Cannot
transit it to OFFLINE.
    at
org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399)
    at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394)
    at
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105)
    at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
    at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-07-31 22:22:46,075 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 60000
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.HMaster$2:
node3,60000,1375322220614-BalancerChore exiting
2013-07-31 22:22:46,075 INFO org.apache.hadoop.hbase.master.CatalogJanitor:
node3,60000,1375322220614-CatalogJanitor exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server listener on 60000
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 2 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 1 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC
Server handler 0 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: exiting
2013-07-31 22:22:46,076 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: exiting
2013-07-31 22:22:46,077 INFO
org.apache.hadoop.hbase.master.cleaner.HFileCleaner:
master-node3,60000,1375322220614.archivedHFileCleaner exiting
2013-07-31 22:22:46,077 INFO
org.apache.hadoop.hbase.master.cleaner.LogCleaner:
master-node3,60000,1375322220614.oldLogCleaner exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.hbase.master.HMaster:
Stopping infoServer
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: exiting
2013-07-31 22:22:46,077 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
IPC Server Responder
2013-07-31 22:22:46,078 INFO org.mortbay.log: Stopped
SelectChannelConnector@0.0.0.0:60010
2013-07-31 22:22:46,127 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 22:22:46,127 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 22:22:46,181 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
aff4d1d8bf470458bb19525e8aef0759 not found on server
node2,60020,1375319046072; failed processing
2013-07-31 22:22:46,181 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
aff4d1d8bf470458bb19525e8aef0759 from server node2,60020,1375319046072 but
it doesn't exist anymore, probably already processed its split
2013-07-31 22:22:46,193 ERROR
org.apache.hadoop.hbase.executor.ExecutorService: Cannot submit
[ClosedRegionHandler-node3,60000,1375322220614-179] because the executor is
missing. Is this process shutting down?
2013-07-31 22:22:46,250 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
28328fdb7181cbd9cc4d6814775e8895 not found on server
node4,60020,1375319042033; failed processing
2013-07-31 22:22:46,250 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
28328fdb7181cbd9cc4d6814775e8895 from server node4,60020,1375319042033 but
it doesn't exist anymore, probably already processed its split
2013-07-31 22:22:46,262 INFO
org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor:
node3,60000,1375322220614.splitLogManagerTimeoutMonitor exiting
2013-07-31 22:22:46,293 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
270a9c371fcbe9cd9a04986e0b77d16b not found on server
node7,60020,1375319044055; failed processing
2013-07-31 22:22:46,293 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
270a9c371fcbe9cd9a04986e0b77d16b from server node7,60020,1375319044055 but
it doesn't exist anymore, probably already processed its split
2013-07-31 22:22:46,294 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Closed zookeeper sessionid=0x240024f5666144b
2013-07-31 22:22:46,361 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Region
aff4d1d8bf470458bb19525e8aef0759 not found on server
node2,60020,1375319046072; failed processing
2013-07-31 22:22:46,362 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Received SPLIT for region
aff4d1d8bf470458bb19525e8aef0759 from server node2,60020,1375319046072 but
it doesn't exist anymore, probably already processed its split
2013-07-31 22:22:46,388 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimeoutMonitor:
node3,60000,1375322220614.timeoutMonitor exiting
2013-07-31 22:22:46,388 INFO
org.apache.hadoop.hbase.master.AssignmentManager$TimerUpdater:
node3,60000,1375322220614.timerUpdater exiting
2013-07-31 22:22:46,402 INFO org.apache.hadoop.hbase.master.HMaster:
HMaster main thread exiting
2013-07-31 22:22:46,402 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:160)
    at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2100)

Seems that HBCK can't do anything. I will start to look at the files into
HDFS, but suggestions are welcome.

JM

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message