hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Lehene <cleh...@adobe.com>
Subject Hbase corrupts data after reporting MSG_REPORT_CLOSE to master during compaction and split process
Date Tue, 02 Sep 2008 11:37:48 GMT
Hi

We managed to end up with two regions starting with the same key and the inability to retrieve
data back from HBase .
I used the RowCounter to count the lines in a table and found corrupted data.

Just to summarize our learning from the following logs.
During a compaction the region server gets a close message from the master and then starts
the split and reports the region as closed (during split). The master gets the MSG_REPORT_CLOSE
(wrongly since the region is closed for a SPLIT) and assigns the same region to a different
server that splits it again.

The detailed steps:
MASTER is HMaster running on 10.131.237.52
REGION is a HRegion running on 10.131.237.51

MASTER: The master assigns the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 region
to 10.131.237.51 (M1)
REGION:  Reports the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 region as available
(R1, M2)
REGION:  Starts to compact the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 region
(R2)
MASTER: Sends a message to 10.131.237.51 to CLOSE the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
region (R3)
REGION: Finishes the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 region compaction
(R4)
REGION: Starts region SPLIT and (by design) CLOSES the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
region (R5, R6)
MASTER: Sends a message to 10.131.237.51 to CLOSE the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
region (R7)
REGION: Reports the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 region as CLOSED
to the master. BUT NOTE THAT THIS IS DUE TO THE SPLIT (M4)
MASTER: Assigns the users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 region to 10.131.237.55
REGION: reports new users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 SPLIT to master
(M7, R9)
MASTER: Assigns new users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 split 231 to
10.131.237.52 (M8)
REGION: closes new users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 split (R10)

Region server 10.131.237.55 also splits the region and we get with 2 regions starting with
the same key (M9)
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454

MASTER: deletes original users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 (M14)



HMaster 10.131.237.52 log

grep "users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6" /usr/local/hbase/logs/hbase-hadoop-master-dev2.log.2008-08-29
2008-08-29 07:58:59,660 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server 10.131.237.54:60020
2008-08-29 07:59:02,728 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.54:60020
2008-08-29 07:59:02,881 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.54:60020
2008-08-29 07:59:02,881 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
open on 10.131.237.54:60020
2008-08-29 07:59:02,881 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region .META.,,1 with startcode
1220014953935 and server 10.131.237.54:60020
M1 2008-08-29 08:50:18,558 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server 10.131.237.51:60020
2008-08-29 08:50:21,935 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:50:25,086 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
...
2008-08-29 08:53:06,020 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:53:06,020 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
M2 2008-08-29 08:53:36,875 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:53:36,876 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
open on 10.131.237.51:60020
M3 2008-08-29 08:53:36,876 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region .META.,,1 with startcode
1220014954000 and server 10.131.237.51:60020
2008-08-29 08:54:12,249 INFO org.apache.hadoop.hbase.master.RegionManager: Skipping region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because it is already closing.
2008-08-29 08:54:15,525 INFO org.apache.hadoop.hbase.master.RegionManager: Skipping region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because it is already closing.
2008-08-29 08:54:18,581 INFO org.apache.hadoop.hbase.master.RegionManager: Skipping region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because it is already closing.
M4 2008-08-29 08:54:24,616 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
M5 2008-08-29 08:54:24,937 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server 10.131.237.55:60020
2008-08-29 08:54:28,133 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.55:60020
2008-08-29 08:54:28,134 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.55:60020
2008-08-29 08:54:28,221 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
open on 10.131.237.55:60020
M6 2008-08-29 08:54:28,221 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region .META.,,1 with startcode
1220014319248 and server 10.131.237.55:60020
M7 2008-08-29 08:54:30,775 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@4861ef74 from 10.131.237.51:60020
M8 2008-08-29 08:54:33,356 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server 10.131.237.52:60020
2008-08-29 08:54:36,419 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:54:39,498 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
...
2008-08-29 08:55:03,727 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
M9 2008-08-29 08:55:04,835 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@62927d48 from 10.131.237.55:60020
2008-08-29 08:55:06,898 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
...
2008-08-29 08:56:04,627 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
M10 2008-08-29 08:56:04,627 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:56:04,642 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
open on 10.131.237.52:60020
M11 2008-08-29 08:56:04,642 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region .META.,,1 with startcode
1220014954468 and server 10.131.237.52:60020
2008-08-29 08:56:10,645 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:56:16,111 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
open on 10.131.236.91:60020
M12 2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region .META.,,1 with startcode
1220014953772 and server 10.131.236.91:60020
2008-08-29 08:56:43,935 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 to server 10.131.236.91:60020
2008-08-29 08:56:47,158 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:50,274 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:53,421 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:53,422 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:53,500 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454
open on 10.131.236.91:60020
M13 2008-08-29 08:56:53,500 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 in region .META.,,1 with startcode
1220014953772 and server 10.131.236.91:60020
2008-08-29 08:57:02,485 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.236.91:60020
2008-08-29 08:57:04,188 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
open on 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region .META.,,1 with startcode
1220024985751 and server 10.131.237.53:60020
2008-08-29 09:00:45,252 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
because daughter splits no longer hold references
[hadoop@ice-dev2-buc ~]$ grep "users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6" /usr/local/hbase/logs/hbase-hadoop-master-ice-dev2-buc.corp.adobe.com.log.2008-08-29
2008-08-29 07:58:59,660 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server 10.131.237.54:60020
2008-08-29 07:59:02,728 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.54:60020
2008-08-29 07:59:02,881 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.54:60020
2008-08-29 07:59:02,881 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
open on 10.131.237.54:60020
2008-08-29 07:59:02,881 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region .META.,,1 with startcode
1220014953935 and server 10.131.237.54:60020
2008-08-29 08:50:18,558 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server 10.131.237.51:60020
2008-08-29 08:50:21,935 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:50:25,086 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
...
2008-08-29 08:53:06,020 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:53:36,875 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:53:36,876 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
open on 10.131.237.51:60020
2008-08-29 08:53:36,876 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region .META.,,1 with startcode
1220014954000 and server 10.131.237.51:60020
2008-08-29 08:54:12,249 INFO org.apache.hadoop.hbase.master.RegionManager: Skipping region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because it is already closing.
2008-08-29 08:54:15,525 INFO org.apache.hadoop.hbase.master.RegionManager: Skipping region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because it is already closing.
2008-08-29 08:54:18,581 INFO org.apache.hadoop.hbase.master.RegionManager: Skipping region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because it is already closing.
2008-08-29 08:54:24,616 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.51:60020
2008-08-29 08:54:24,937 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server 10.131.237.55:60020
2008-08-29 08:54:28,133 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.55:60020
2008-08-29 08:54:28,134 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from 10.131.237.55:60020
2008-08-29 08:54:28,221 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
open on 10.131.237.55:60020
2008-08-29 08:54:28,221 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region .META.,,1 with startcode
1220014319248 and server 10.131.237.55:60020
2008-08-29 08:54:30,775 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@4861ef74 from 10.131.237.51:60020
2008-08-29 08:54:33,356 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server 10.131.237.52:60020
2008-08-29 08:54:36,419 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:54:39,498 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
...
2008-08-29 08:55:03,727 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:55:04,835 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_SPLIT:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@62927d48 from 10.131.237.55:60020
2008-08-29 08:55:06,898 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
...
2008-08-29 08:56:04,627 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:56:04,627 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:56:04,642 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
open on 10.131.237.52:60020
2008-08-29 08:56:04,642 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region .META.,,1 with startcode
1220014954468 and server 10.131.237.52:60020
2008-08-29 08:56:10,645 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.52:60020
2008-08-29 08:56:16,111 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
open on 10.131.236.91:60020
2008-08-29 08:56:19,134 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region .META.,,1 with startcode
1220014953772 and server 10.131.236.91:60020
2008-08-29 08:56:43,935 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 to server 10.131.236.91:60020
2008-08-29 08:56:47,158 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:50,274 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:53,421 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:53,422 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from 10.131.236.91:60020
2008-08-29 08:56:53,500 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454
open on 10.131.236.91:60020
2008-08-29 08:56:53,500 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 in region .META.,,1 with startcode
1220014953772 and server 10.131.236.91:60020
2008-08-29 08:57:02,485 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.236.91:60020
2008-08-29 08:57:04,188 INFO org.apache.hadoop.hbase.master.RegionManager: assigning region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_PROCESS_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ServerManager: Received MSG_REPORT_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
open on 10.131.237.53:60020
2008-08-29 08:57:07,309 INFO org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating
row users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region .META.,,1 with startcode
1220024985751 and server 10.131.237.53:60020
M14 2008-08-29 09:00:45,252 INFO org.apache.hadoop.hbase.master.BaseScanner: Deleting region
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 because daughter splits no longer
hold references


Region server log 10.131.237.51

grep users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6 /usr/local/hbase/logs/hbase-hadoop-regionserver-dev1.log.2008-08-29
2008-08-29 08:50:18,868 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
2008-08-29 08:52:52,323 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
R1 2008-08-29 08:53:34,330 INFO org.apache.hadoop.hbase.regionserver.HRegion: region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774/62578664
available
R2 2008-08-29 08:53:34,330 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
R3 2008-08-29 08:54:09,035 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@284f4565
R4 2008-08-29 08:54:24,224 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed
on region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in 49sec
R5 2008-08-29 08:54:24,224 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split
of region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
R6 2008-08-29 08:54:24,234 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
R7 2008-08-29 08:54:24,393 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@284f4565
R8 2008-08-29 08:54:24,393 WARN org.apache.hadoop.hbase.regionserver.HRegion: region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
already closed
R9 2008-08-29 08:54:27,323 INFO org.apache.hadoop.hbase.regionserver.HRegion: region users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231/620189721
available
R10 2008-08-29 08:54:27,324 INFO org.apache.hadoop.hbase.regionserver.HRegion: closed users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
R11 2008-08-29 08:54:27,726 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread:
region split, META updated, and report to master all successful. Old region=REGION => {NAME
=> 'users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774', STARTKEY => '27B5C924-6FE4-4484-B131-D5C9E71AF4A6',
ENDKEY => '2FB0249D-0A95-47BB-A4F1-6C7600039542', ENCODED => 62578664, OFFLINE =>
true, SPLIT => true, TABLE => {{NAME => 'users', IS_ROOT => 'false', IS_META =>
'false', FAMILIES => [{NAME => 'test', BLOOMFILTER => 'false', COMPRESSION =>
'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false',
BLOCKCACHE => 'false'}]}}, new regions: users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231,
users,2BB6AC25-2EA6-4311-9670-49B68B7BB975,1220025264231. Split took 3sec



Mime
View raw message