hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrea <andrea.gandini...@gmail.com>
Subject Re: HBase and HDFS replication
Date Wed, 04 Dec 2013 11:18:56 GMT
I'm sorry for the delay but in the past days I had troubles with my amazon 
account! Anyway, I have this situation (status 'simple' output) with an hdfs 
replication set at 3:

3 live servers
    ip-10-254-104-164:60020 1386152974983
        requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=55, 
maxHeapMB=6483
    ip-10-232-23-30:60020 1386152980000
        requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=69, 
maxHeapMB=6483
    ip-10-253-6-188:60020 1386152977755
        requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=1084, 
maxHeapMB=6483
0 dead servers
Aggregate load: 0, regions: 8

/////////////////////////////////////////////////////////////////////////

I kill the node ip-10-254-104-164 (the first one) and here the situation:

2 live servers
    ip-10-232-23-30:60020 1386152980000
        requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=39, 
maxHeapMB=6483
    ip-10-253-6-188:60020 1386152977755
        requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=1087, 
maxHeapMB=6483
1 dead servers
    ip-10-254-104-164,60020,1386152974983
Aggregate load: 0, regions: 6

/////////////////////////////////////////////////////////////////////////

Note that at now (after more than 30 minutes it doesn't recover the 
regions!!!)

And here the master output (the first line is the master adding the killed 
node to the dead server):

2013-12-04 10:44:01,907 DEBUG org.apache.hadoop.hbase.master.ServerManager: 
Added=ip-10-254-104-164,60020,1386152974983 to dead servers, submitted 
shutdown handler to be executed, root=false, meta=false
2013-12-04 10:44:01,907 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs 
for ip-10-254-104-164,60020,1386152974983
2013-12-04 10:44:01,914 DEBUG 
org.apache.hadoop.hbase.master.MasterFileSystem: Renamed region directory: 
hdfs://master1:54310/hbase/.logs/ip-10-254-104-164,60020,1386152974983-
splitting
2013-12-04 10:44:01,914 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
dead splitlog workers [ip-10-254-104-164,60020,1386152974983]
2013-12-04 10:44:01,916 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager: Scheduling batch of logs to 
split
2013-12-04 10:44:01,918 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
started splitting logs in [hdfs://master1:54310/hbase/.logs/ip-10-254-104-
164,60020,1386152974983-splitting]
2013-12-04 10:44:01,926 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager: put up splitlog task at 
znode /hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-
254-104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541
2013-12-04 10:44:01,928 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired 
/hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-254-
104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541 ver = 0
2013-12-04 10:44:01,935 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
task /hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-
254-104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541 acquired by ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:44:02,403 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 
0
2013-12-04 10:44:03,401 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 
0
2013-12-04 10:44:04,401 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 
0
2013-12-04 10:44:04,980 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
task /hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-
254-104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541 entered state done ip-10-253-
6-188,60020,1386152977755
2013-12-04 10:44:04,997 DEBUG 
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived processed 
log hdfs://master1:54310/hbase/.logs/ip-10-254-104-164,60020,1386152974983-
splitting/ip-10-254-104-164%2C60020%2C1386152974983.1386152986541 to 
hdfs://master1:54310/hbase/.oldlogs/ip-10-254-104-
164%2C60020%2C1386152974983.1386152986541
2013-12-04 10:44:04,999 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
Done splitting 
/hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-254-
104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541
2013-12-04 10:44:05,003 INFO org.apache.hadoop.hbase.master.SplitLogManager: 
finished splitting (more than or equal to) 0 bytes in 1 log files in 
[hdfs://master1:54310/hbase/.logs/ip-10-254-104-164,60020,1386152974983-
splitting] in 3085ms
2013-12-04 10:44:05,004 DEBUG 
org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted 
/hbase/splitlog/hdfs%3A%2F%2Fmaster1%3A54310%2Fhbase%2F.logs%2Fip-10-254-
104-164%2C60020%2C1386152974983-splitting%2Fip-10-254-104-
164%252C60020%252C1386152974983.1386152986541
2013-12-04 10:44:05,006 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Creating scanner over .META. starting at key ''
2013-12-04 10:44:05,006 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Advancing internal scanner to startKey at ''
2013-12-04 10:44:05,013 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => 
'', ENCODED => 1028785192,}
2013-12-04 10:44:05,016 DEBUG 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined and 
split region usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.; 
checking daughter presence
2013-12-04 10:44:05,019 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Creating scanner over .META. starting at key 
'usertable,,1386153580623.655f3644e994b0cc0e7ce67c10f4cc98.'
2013-12-04 10:44:05,019 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Advancing internal scanner to startKey at 
'usertable,,1386153580623.655f3644e994b0cc0e7ce67c10f4cc98.'
2013-12-04 10:44:05,023 DEBUG 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter 
usertable,,1386153580623.655f3644e994b0cc0e7ce67c10f4cc98. present
2013-12-04 10:44:05,023 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Creating scanner over .META. starting at key 
'usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d36
745a.'
2013-12-04 10:44:05,023 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Advancing internal scanner to startKey at 
'usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d36
745a.'
2013-12-04 10:44:05,027 DEBUG 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter 
usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d367
45a. present
2013-12-04 10:44:05,027 DEBUG 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Offlined and 
split region 
usertable,user306830786865523530,1386153580623.0b0a5ec5b6e069f385b38d658d367
45a.; checking daughter presence
2013-12-04 10:44:05,029 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Creating scanner over .META. starting at key 
'usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6
eb0a.'
2013-12-04 10:44:05,029 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Advancing internal scanner to startKey at 
'usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6
eb0a.'
2013-12-04 10:44:05,033 DEBUG 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. present
2013-12-04 10:44:05,033 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Creating scanner over .META. starting at key 
'usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c35
2dd1a.'
2013-12-04 10:44:05,033 DEBUG org.apache.hadoop.hbase.client.ClientScanner: 
Advancing internal scanner to startKey at 
'usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c35
2dd1a.'
2013-12-04 10:44:05,036 DEBUG 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Daughter 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. present
2013-12-04 10:44:05,037 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 10:44:05,042 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one; 
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2, 
available=2) available servers
2013-12-04 10:44:05,042 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:44:05,057 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:44:05,057 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:05,068 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one; 
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2, 
available=2) available servers
2013-12-04 10:44:05,068 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 10:44:05,079 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:05,081 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: 2 regions 
which were planned to open on ip-10-254-104-164,60020,1386152974983 have 
been re-assigned.
2013-12-04 10:44:05,081 INFO 
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Finished 
processing of shutdown of ip-10-254-104-164,60020,1386152974983
2013-12-04 10:44:05,099 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:05,116 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:11,771 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:11,786 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:15,109 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:15,124 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:18,447 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:18,463 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:21,784 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:21,800 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:25,122 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:25,138 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:28,467 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:28,476 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:31,805 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:31,813 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:35,141 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:35,151 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:38,485 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:38,490 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:41,824 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:41,829 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:45,162 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:45,166 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:48,501 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:48,506 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:51,841 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:51,845 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OPENING, ts=1386153891924, 
server=ip-10-232-23-30,60020,1386152980000, 
797ead45993e04d649da99d4c352dd1a=usertable,user41007679046964...
2013-12-04 10:44:53,748 DEBUG org.apache.hadoop.hbase.client.MetaScanner: 
Scanning .META. starting at row= for max=2147483647 rows using 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@
4bc900b6
2013-12-04 10:44:53,783 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: 
Deleting region usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425. 
because daughter splits no longer hold references
2013-12-04 10:44:53,784 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
ARCHIVING region 
hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425
2013-12-04 10:44:53,784 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Have an archive directory, preparing to move files
2013-12-04 10:44:53,789 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Archiving:[class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, 
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily]
2013-12-04 10:44:53,789 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Starting to archive files:[class 
org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, 
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily]
2013-12-04 10:44:53,789 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
moving files to the archive directory: 
hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22a774
25
2013-12-04 10:44:53,793 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Created archive 
directory:hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d
23ac22a77425
2013-12-04 10:44:53,793 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, 
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily
2013-12-04 10:44:53,794 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, 
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily is a directory, archiving children files
2013-12-04 10:44:53,796 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
moving files to the archive directory: 
hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22a774
25/family
2013-12-04 10:44:53,800 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Created archive 
directory:hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d
23ac22a77425/family
2013-12-04 10:44:53,800 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, 
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily/8331259e07d7457ab61961cd629ea1f8
2013-12-04 10:44:53,802 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
No existing file in archive 
for:hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22
a77425/family/8331259e07d7457ab61961cd629ea1f8, free to archive original 
file.
2013-12-04 10:44:53,805 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Finished archiving file from: class 
org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, 
file:hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425/f
amily/8331259e07d7457ab61961cd629ea1f8, to: 
hdfs://master1:54310/hbase/.archive/usertable/05a29854894d8bed8c6d23ac22a774
25/family/8331259e07d7457ab61961cd629ea1f8
2013-12-04 10:44:53,805 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Successfully resolved and archived, now can just delete region.
2013-12-04 10:44:53,808 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: 
Deleted all region files in: 
hdfs://master1:54310/hbase/usertable/05a29854894d8bed8c6d23ac22a77425
2013-12-04 10:44:55,179 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:55,188 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:44:58,517 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:44:58,526 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:01,858 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:01,865 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:05,194 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:05,202 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:08,533 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:08,540 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:11,871 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:11,878 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:15,209 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:15,216 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:18,547 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:18,554 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:21,885 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:21,899 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:25,224 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:25,237 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:28,562 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:28,575 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:31,900 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:31,913 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:35,238 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:35,251 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:38,576 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:38,590 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:41,914 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:41,928 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,257 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-232-23-
30,60020,1386152980000, region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,258 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,258 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one; 
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2, 
available=1) available servers
2013-12-04 10:45:45,259 DEBUG 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED 
event for 296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,259 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; 
was=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a. state=CLOSED, ts=1386153945345, server=ip-10-232-23-
30,60020,1386152980000
2013-12-04 10:45:45,259 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 10:45:45,265 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,266 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,266 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan 
for region 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a.; 
plan=hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d8
22f5c0d6eb0a., src=, dest=ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,266 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,279 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-253-6-
188,60020,1386152977755, region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,280 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:45:45,280 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one; 
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2, 
available=1) available servers
2013-12-04 10:45:45,280 DEBUG 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED 
event for 797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,281 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; 
was=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a. state=CLOSED, ts=1386153945334, server=ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:45:45,281 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:45:45,283 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,289 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,289 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,289 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan 
for region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a.; 
plan=hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649d
a99d4c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,289 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:45:45,303 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:45,333 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:45,336 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:52,004 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:52,008 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:55,342 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:55,345 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:45:58,680 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:45:58,686 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:02,019 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:02,024 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:05,358 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:05,361 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:08,704 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:08,707 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:12,043 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:12,046 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:15,387 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:15,389 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:18,725 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:18,728 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:22,064 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:22,067 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:25,406 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:25,407 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:28,742 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:28,746 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:32,081 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:32,090 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:35,420 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:35,428 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:38,758 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:38,765 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:42,095 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:42,103 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:45,366 WARN 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Failed all from region=.META.,,1.1028785192, hostname=ip-10-232-23-30, 
port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: 
Call to ip-10-232-23-30/10.232.23.30:60020 failed on socket timeout 
exception: java.net.SocketTimeoutException: 60000 millis timeout while 
waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34262 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatchCallback(HConnectionManager.java:1711)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatch(HConnectionManager.java:1563)
	at org.apache.hadoop.hbase.client.HTable.delete(HTable.java:801)
	at 
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:146)
	at 
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:133)
	at 
org.apache.hadoop.hbase.catalog.MetaEditor.deleteRegion(MetaEditor.java:310)
	at 
org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.jav
a:242)
	at 
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:145)
	at 
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:93)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34262 remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
	at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
	at com.sun.proxy.$Proxy11.multi(Unknown Source)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1540)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1538)
	at 
org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.
java:228)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1547)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1535)
	at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
	... 1 more
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while 
waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34262 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:46:45,371 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Removed .META.,,1.1028785192 for tableName=.META. from cache because of 
usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.
2013-12-04 10:46:45,372 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Retry 1, sleep for 1009ms!
2013-12-04 10:46:45,434 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:45,443 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:46,383 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:46:46,386 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Cached location for .META.,,1.1028785192 is ip-10-232-23-30:60020
2013-12-04 10:46:48,773 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:48,780 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:52,111 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:52,186 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:55,449 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:55,523 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:46:58,787 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:46:58,869 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:02,125 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:02,208 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:05,464 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:05,546 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:08,802 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:08,885 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:12,140 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:12,223 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:15,478 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:15,561 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:18,816 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:18,899 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:22,154 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:22,237 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-232-23-30,60020,1386152980000, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,505 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-253-6-
188,60020,1386152977755, region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:25,505 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,505 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one; 
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2, 
available=1) available servers
2013-12-04 10:47:25,505 DEBUG 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED 
event for 296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:25,505 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; 
was=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a. state=CLOSED, ts=1386154045562, server=ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:47:25,505 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 10:47:25,514 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 10:47:25,514 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,514 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan 
for region 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a.; 
plan=hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d8
22f5c0d6eb0a., src=, dest=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,514 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,579 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-232-23-
30,60020,1386152980000, region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,580 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:25,580 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one; 
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2, 
available=1) available servers
2013-12-04 10:47:25,580 DEBUG 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED 
event for 797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,580 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; 
was=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a. state=CLOSED, ts=1386154045672, server=ip-10-232-23-
30,60020,1386152980000
2013-12-04 10:47:25,580 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:47:25,587 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,587 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,587 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan 
for region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a.; 
plan=hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649d
a99d4c352dd1a., src=, dest=ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,587 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 10:47:25,596 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:25,628 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:28,968 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:32,306 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:35,644 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:38,982 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:42,320 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:45,657 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:46,438 WARN 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Failed all from region=.META.,,1.1028785192, hostname=ip-10-232-23-30, 
port=60020
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: 
Call to ip-10-232-23-30/10.232.23.30:60020 failed on socket timeout 
exception: java.net.SocketTimeoutException: 60000 millis timeout while 
waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
	at java.util.concurrent.FutureTask.get(FutureTask.java:111)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatchCallback(HConnectionManager.java:1711)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
processBatch(HConnectionManager.java:1563)
	at org.apache.hadoop.hbase.client.HTable.delete(HTable.java:801)
	at 
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:146)
	at 
org.apache.hadoop.hbase.catalog.MetaEditor.deleteFromMetaTable(MetaEditor.ja
va:133)
	at 
org.apache.hadoop.hbase.catalog.MetaEditor.deleteRegion(MetaEditor.java:310)
	at 
org.apache.hadoop.hbase.master.CatalogJanitor.cleanParent(CatalogJanitor.jav
a:242)
	at 
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:145)
	at 
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:93)
	at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34267 remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
	at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
	at com.sun.proxy.$Proxy11.multi(Unknown Source)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1540)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3$1.call(HConnectionManager.java:1538)
	at 
org.apache.hadoop.hbase.client.ServerCallable.withoutRetries(ServerCallable.
java:228)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1547)
	at 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$
3.call(HConnectionManager.java:1535)
	at 
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
	... 1 more
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while 
waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:47:46,440 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Removed .META.,,1.1028785192 for tableName=.META. from cache because of 
usertable,,1386153335227.05a29854894d8bed8c6d23ac22a77425.
2013-12-04 10:47:46,440 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Retry 2, sleep for 1003ms!
2013-12-04 10:47:47,446 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:47:48,996 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:52,333 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:55,671 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:47:59,010 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:02,359 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:05,697 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:09,035 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:12,373 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:15,710 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:19,048 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:22,388 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:25,726 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:29,064 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:32,403 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:35,742 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:39,081 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:42,418 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:45,756 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:46,454 WARN 
org.apache.hadoop.hbase.master.AssignmentManager: Call openRegion() to ip-
10-232-23-30,60020,1386152980000 has timed out when trying to assign 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a., but the region might already be opened on ip-10-232-23-
30,60020,1386152980000.
java.net.SocketTimeoutException: Call to ip-10-232-23-30/10.232.23.30:60020 
failed on socket timeout exception: java.net.SocketTimeoutException: 60000 
millis timeout while waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
	at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
	at com.sun.proxy.$Proxy11.openRegion(Unknown Source)
	at 
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.ja
va:533)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1729)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1426)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1398)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1393)
	at 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedReg
ionHandler.java:105)
	at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while 
waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:48:49,094 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:52,438 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:55,776 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:48:59,114 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:02,452 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,797 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_FAILED_OPEN, server=ip-10-253-6-
188,60020,1386152977755, region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,797 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-253-6-188,60020,1386152977755
2013-12-04 10:49:05,797 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. so generated a random one; 
hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000; 2 (online=2, 
available=1) available servers
2013-12-04 10:49:05,797 DEBUG 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED 
event for 797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,797 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; 
was=usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4
c352dd1a. state=CLOSED, ts=1386154145862, server=ip-10-253-6-
188,60020,1386152977755
2013-12-04 10:49:05,797 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
797ead45993e04d649da99d4c352dd1a with OFFLINE state
2013-12-04 10:49:05,803 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407, 
region=797ead45993e04d649da99d4c352dd1a
2013-12-04 10:49:05,804 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:05,804 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan 
for region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a.; 
plan=hri=usertable,user4100767904696415789,1386153640366.797ead45993e04d649d
a99d4c352dd1a., src=, dest=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:05,804 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a. to ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:46,510 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=0 of 140 
failed; retrying after sleep of 1001 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34267 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:49:46,514 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:47,517 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:49:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505, 
server=null, 
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 10:50:46,570 WARN 
org.apache.hadoop.hbase.master.AssignmentManager: Call openRegion() to ip-
10-232-23-30,60020,1386152980000 has timed out when trying to assign 
usertable,user4100767904696415789,1386153640366.797ead45993e04d649da99d4c352
dd1a., but the region might already be opened on ip-10-232-23-
30,60020,1386152980000.
java.net.SocketTimeoutException: Call to ip-10-232-23-30/10.232.23.30:60020 
failed on socket timeout exception: java.net.SocketTimeoutException: 60000 
millis timeout while waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1035)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1008)
	at 
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngi
ne.java:86)
	at com.sun.proxy.$Proxy11.openRegion(Unknown Source)
	at 
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.ja
va:533)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1729)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1426)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1398)
	at 
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.ja
va:1393)
	at 
org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedReg
ionHandler.java:105)
	at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while 
waiting for channel to be ready for read. ch : 
java.nio.channels.SocketChannel[connected local=/10.232.25.198:34267 
remote=ip-10-232-23-30/10.232.23.30:60020]
	at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
	at 
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
	at java.io.FilterInputStream.read(FilterInputStream.java:133)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBas
eClient.java:373)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClie
nt.java:652)
	at 
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:586)
2013-12-04 10:51:46,580 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=1 of 140 
failed; retrying after sleep of 1004 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34267 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:51:46,583 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:51:47,589 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:52:47,648 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=2 of 140 
failed; retrying after sleep of 1002 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34279 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:52:47,651 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:52:48,659 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:53:48,679 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=3 of 140 
failed; retrying after sleep of 2011 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34281 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:53:48,681 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:53:50,688 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:54:50,735 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=4 of 140 
failed; retrying after sleep of 2003 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34286 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:54:50,738 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:54:52,758 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:54:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505, 
server=null, 
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 10:55:52,790 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=5 of 140 
failed; retrying after sleep of 4037 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34290 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:55:52,793 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:55:56,812 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:56:56,872 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=6 of 140 
failed; retrying after sleep of 4023 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34295 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:56:56,874 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:57:00,883 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:58:00,917 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=7 of 140 
failed; retrying after sleep of 8059 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34300 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:58:00,920 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:58:08,944 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:59:09,003 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=8 of 140 
failed; retrying after sleep of 16038 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34304 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 10:59:09,005 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:59:25,061 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 10:59:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505, 
server=null, 
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:00:25,094 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=9 of 140 
failed; retrying after sleep of 32233 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34308 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:00:25,097 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:00:57,187 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:01:57,240 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=10 of 140 
failed; retrying after sleep of 64260 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34313 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:01:57,242 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:03:01,712 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:04:01,765 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=11 of 140 
failed; retrying after sleep of 64281 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34318 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:04:01,768 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:04:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505, 
server=null, 
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:05:06,257 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:06:06,273 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=12 of 140 
failed; retrying after sleep of 64504 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34328 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:06:06,276 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:07:10,650 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:08:10,659 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=13 of 140 
failed; retrying after sleep of 64573 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34334 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:08:10,662 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:09:14,773 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:09:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505, 
server=null, 
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:10:14,832 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=14 of 140 
failed; retrying after sleep of 64639 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34342 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:10:14,834 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:11:19,316 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:12:19,350 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=15 of 140 
failed; retrying after sleep of 64320 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34350 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:12:19,352 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:13:23,817 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:14:23,876 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=16 of 140 
failed; retrying after sleep of 64241 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34358 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:14:23,878 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:14:53,740 DEBUG org.apache.hadoop.hbase.master.HMaster: Not 
running balancer because 2 region(s) in transition: 
{296a4e3f4e890d3f76d822f5c0d6eb0a=usertable,user306830786865523530,138615364
0366.296a4e3f4e890d3f76d822f5c0d6eb0a. state=OFFLINE, ts=1386154045505, 
server=null, 
797ead45993e04d649da99d4c352dd1a=usertable,user4100767904696415789,138615364
0366.797ead45993...
2013-12-04 11:15:28,187 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:16:28,246 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-
,,0.70236052, hostname=ip-10-232-23-30, port=60020}, attempt=17 of 140 
failed; retrying after sleep of 64418 because: Call to ip-10-232-23-
30/10.232.23.30:60020 failed on socket timeout exception: 
java.net.SocketTimeoutException: 60000 millis timeout while waiting for 
channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/10.232.25.198:34367 remote=ip-10-232-23-30/10.232.23.30:60020]
2013-12-04 11:16:28,248 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:17:30,592 INFO 
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition 
timed out:  
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. state=OFFLINE, ts=1386154045505, server=null
2013-12-04 11:17:30,593 INFO 
org.apache.hadoop.hbase.master.AssignmentManager: Region has been OFFLINE 
for too long, reassigning 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to a random server
2013-12-04 11:17:30,594 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: 
master:60000-0x142bd264a940000 Creating (or updating) unassigned node for 
296a4e3f4e890d3f76d822f5c0d6eb0a with OFFLINE state
2013-12-04 11:17:30,601 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=M_ZK_REGION_OFFLINE, server=ip-10-232-25-198,60000,1386152972407, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:30,601 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. destination server is ip-10-232-23-30,60020,1386152980000
2013-12-04 11:17:30,601 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: No previous transition 
plan was found (or we are ignoring an existing plan) for 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. so generated a random one; 
hri=usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c
0d6eb0a., src=, dest=ip-10-253-6-188,60020,1386152977755; 2 (online=2, 
available=2) available servers
2013-12-04 11:17:30,601 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region 
usertable,user306830786865523530,1386153640366.296a4e3f4e890d3f76d822f5c0d6e
b0a. to ip-10-253-6-188,60020,1386152977755
2013-12-04 11:17:30,612 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:30,642 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:32,253 DEBUG 
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: 
Looked up root region location, 
connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
ementation@4bc900b6; serverName=ip-10-232-23-30,60020,1386152980000
2013-12-04 11:17:37,313 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:40,651 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:43,988 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:47,325 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:50,669 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:54,007 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a
2013-12-04 11:17:57,345 DEBUG 
org.apache.hadoop.hbase.master.AssignmentManager: Handling 
transition=RS_ZK_REGION_OPENING, server=ip-10-253-6-188,60020,1386152977755, 
region=296a4e3f4e890d3f76d822f5c0d6eb0a


Mime
View raw message