hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Lehene <cleh...@adobe.com>
Subject Re: Hbase corrupts data after reporting MSG_REPORT_CLOSE to master during compaction and split process
Date Tue, 09 Sep 2008 14:24:32 GMT
Hi,

I managed to reproduce the corruption and also have full debug logs, but first I'll explain the whys and hows of the bug and also how I think it can be fixed.
( I'm going to send my takeaways on how we managed to insert 300GB in less then 6 hours on a 5 node cluster and also some advice/issues in another mail.)

Next assumptions are based on understanding the actual code (don't worry if I didn't get them all right, please read the entire mail).

- The master _assigns_ a region to a server by sending a MSG_REGION_OPEN
- On heartbeat region servers report the current load and a list of MLR - most loaded regions (in fact just a list of first N online regions).
- Upon opening a newly assigned region, a region server will try to compact and split that region.
- The region is NOT marked offline when compaction starts
- The region is marked OFFLINE:true, SPLIT:true during a SPLIT

Our scenario goes this way:

Master (M) assigns region A to region server R1
R1 starts compaction and split of A
R1 on heart beat sends it's load and a list of MLR that contains A
M decides to reassign the extra regions and sends a MSG_CLOSE_REGION A to R1
R1 finishes the compaction and splits A into A1 and A2 (A1 has the same start key as A)
M assigns A a to R2
R2 starts compaction and split of A
R2 finishes the compaction and splits A into A_clone_1 and A_clone_2 (A_clone_1 has the same start key as A and IMPORTANT the same start key as A1)

Now we get A1 and A_clone_1 almost identical starting with the same key. Cluster is corrupted. We should care less what happens next. But the ideea is that they are both in .META.

I figured several places where this could be avoided and I'm going to state a few disjoint questions. Both Master and Region could be held responsible in my opinion but I guess it's a matter of architectural philosophy. Please note that any of these question would be a starting point for the fix.

- Why when getting a MSG_CLOSE_REGION A, the region server doesn't abort the current compact split operation to leaving A in the original state and close it immediately?
- Why doesn't a region server DELETE a region after a SPLIT?( I guess it could be offline by then and it's not himself to decide that, but still..)
- Why when assigning a region to a new region server the master doesn't check  the region status? It might be splitting or already split. I guess this would need a new state.
- Why when opening/compacting/splitting a region server doesn't check if the region is OFFLINE:true or SPLIT:true?

I have the logs available, however they are pretty large and I might need to clean them a little, but I could make them available if that's really needed. However I think the scenario and questions might be enough for a bug and a fix.

Thanks,
Cosmin





On 9/4/08 7:57 PM, "Jean-Daniel Cryans" <jdcryans@apache.org> wrote:

Cosmin,

Next time you get a corrupted region, try restarting HBase and Hadoop. It
worked for me.

J-D

On Thu, Sep 4, 2008 at 12:52 PM, stack <stack@duboce.net> wrote:

> Cosmin Lehene wrote:
>
>> ////
>>
>> However I wonder why I didn't get the stack trace.
>>
>>
>
> I wonder too.  I see the LOG.error being passed the exception -- and J-D is
> getting it above.  Whats your JVM?
>
>> And yes, after that it complained it couldn't get a file from HDFS.I
>> haven't gave a thorough look the the HStore.compact methods, but I assumed
>> that when doing the compaction it copies some files to a new file one by one
>> and maybe doesn't finish, however gets in a state when the data is actually
>> in two places and the exception is not treated at the right level, but
>> rather in CompactSplitThread where there isn't much to be done.
>>
>>
>
> If compaction fails, should be no means of it damaging the running
> instance.  The compaction mess is just cleaned up and we retry later.  Only
> on successful compaction does the newly compacted file get moved into place.
>
>  My biggest concern is that I manage to corrupt it in different ways. I do
>> wonder, however if these could be isolated in unit tests so that I could
>> just send a unit test instead.
>>
>>
>
> You could try that but my guess is that it would be hard to reproduce in a
> small test case.  Best would be making your logs available so we could look
> at them.
>
> Thanks Cosmin.
>
> St.Ack
>
>
>
>  Cosmin
>>
>>
>>
>>> This region is now corrupted. We see stuff like that doing a get in the
>>> shell:
>>>
>>> 08/09/04 11:27:44 DEBUG client.HConnectionManager$TableServers: reloading
>>> table servers because: java.io.IOException: Cannot open filename
>>>
>>> /hbase/amsterdam_development/web_pages/798949571/attribute/mapfiles/7908375253
>>> 086018803/data
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1140)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1
>>> 442)
>>>        at
>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1281>
>>> )
>>>        at
>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1385)
>>>        at
>>> org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1337)
>>>        at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>>        at
>>>
>>> org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:18
>>> 47)
>>>        at
>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1877)
>>>        at
>>> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1782)
>>>        at
>>> org.apache.hadoop.io.MapFile$Reader.seekInternal(MapFile.java:463)
>>>        at
>>> org.apache.hadoop.io.MapFile$Reader.getClosest(MapFile.java:558)
>>>        at
>>> org.apache.hadoop.io.MapFile$Reader.getClosest(MapFile.java:541)
>>>        at
>>>
>>> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.getC
>>> losest(HStoreFile.java:757)
>>> ...
>>>
>>> I guess that it happened after the region got this:
>>>
>>> 2008-08-28 18:42:08,142 WARN org.apache.hadoop.dfs.DFSClient:
>>> DataStreamer
>>> Exception: org.apache.hadoop.ipc.RemoteException:
>>> org.apache.hadoop.dfs.LeaseExpiredException: No lease on
>>>
>>> /hbase/amsterdam_development/web_pages/compaction.dir/1338764398/attribute/map
>>> files/5346237486847461749/data
>>> File does not exist. [Lease.  Holder: 44 46 53 43 6c 69 65 6e 74 5f 2d 31
>>> 39
>>> 31 39 31 31 30 38 30 36, heldlocks: 0, pendingcreates: 5]
>>>        at
>>> org.apache.hadoop.dfs.FSNamesystem.checkLease(FSNamesystem.java:1194)
>>>        at
>>>
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1125)
>>>        at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>        at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
>>>        at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
>>> ava:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>
>>>        at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>        at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
>>>        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>>>        at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j
>>> ava:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at
>>>
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocation
>>> Handler.java:82)
>>>        at
>>>
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandle
>>> r.java:59)
>>>        at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient
>>> .java:2335)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClien
>>> t.java:2220)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:170
>>> 2)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.jav
>>> a:1842)
>>>
>>> 2008-08-28 18:42:08,161 WARN org.apache.hadoop.dfs.DFSClient: Error
>>> Recovery
>>> for block blk_-8223656433789893214 bad datanode[0]
>>> 2008-08-28 18:42:08,162 ERROR
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split
>>> failed for region web_pages,
>>>
>>> http://automotive.autoaubaine.com/occasions-auto/Saguenay-Lac-St-Jean-listing
>>> .
>>> html,1219963284600
>>> java.io.IOException: Could not get block locations. Aborting...
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient
>>> .java:2081)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1300(DFSClient.java:170
>>> 2)
>>>        at
>>>
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.jav
>>> a:1818)
>>>
>>>
>>> This seems like a serious problem.
>>>
>>> J-D
>>>
>>> On Thu, Sep 4, 2008 at 11:25 AM, Cosmin Lehene <clehene@adobe.com>
>>> wrote:
>>>
>>>
>>>
>>>>
>>>> On 9/3/08 6:40 PM, "stack" <stack@duboce.net> wrote:
>>>>
>>>>
>>>>
>>>>> Cosmin Lehene wrote:
>>>>>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for explaining things.
>>>>>> HBase Version    0.2.0, r690650
>>>>>>
>>>>>>
>>>>>>
>>>>> Is that the released 0.2.0?
>>>>>
>>>>>
>>>>>
>>>> It was a build from the 0.2 branch.
>>>>
>>>>
>>>>> Hadoop Version    0.17.2, r684969
>>>>>>
>>>>>> It's hard to reproduce that. Right now HDFS gets corrupted after we
>>>>>>
>>>>>>
>>>>> start our
>>>>
>>>>
>>>>> insertion scripts.  I've enabled DEBUG however. I'll retry fresh with
>>>>>> an
>>>>>> upgrade  to hadoop 0.18 and Hbase trunk.
>>>>>>
>>>>>>
>>>>>>
>>>>> I would be interested in seeing logs of a corrupt hbase especially when
>>>>> we're under the illusion that we left 'corruption' long behind us.
>>>>>
>>>>>
>>>>>> However we might start  digging more in the code to see if we can
>>>>>> start
>>>>>> patching things.
>>>>>> Are there any issues with HDFS as well or it's just Hbase that needs
>>>>>>
>>>>>>
>>>>> fixing?
>>>>      I upgraded the cluster to HBase trunk "0.3.0"(I guess this should
>>>> be  0.18)
>>>>  and Hadoop 0.18 release. We're still pushing data, but so far it looks
>>>> better than 0.2.0, however region servers die with
>>>>
>>>> 2008-09-03 10:17:20,457 FATAL
>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Ran out of memory
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> And the data got corrupted again with debug information only on the
>>>> master
>>>> server. Apparently it started a compaction and failed with
>>>> NullPointerException leaving the region corrupted. Unfortunately the
>>>> CompactSplitThread only logs the exception, not the stack trace.
>>>>
>>>> Most of the problems appear when inserting heavily (between 1000 - 2000
>>>> requests/s). Under light insert rate (<400 requests/s) it behaves
>>>> better.
>>>>
>>>> I recompiled now with more logging and started to load data again.
>>>> Hopefully we will cross the 20M boundary.
>>>>
>>>> Cosmin
>>>>
>>>>
>>>>> Usually its HBase.  But if HDFS is sickly, HBase is too.
>>>>>
>>>>> St.Ack
>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Cosmin
>>>>>>
>>>>>> On 9/2/08 10:14 PM, "stack" <stack@duboce.net> wrote:
>>>>>>
>>>>>> Thanks for digging in Cosmin.   What version of hbase?
>>>>>>
>>>>>> FYI, to repair the table, there is a tool to merge adjacent regions:
>>>>>>
>>>>>> $ ./bin/hbase org.apache.hadoop.hbase.util.Merge
>>>>>> Usage: bin/hbase merge <table-name> <region-1> <region-2>
>>>>>>
>>>>>> See below for comments inline:
>>>>>>
>>>>>> Cosmin Lehene wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> ...
>>>>>>> HMaster 10.131.237.52 log
>>>>>>>
>>>>>>> grep "users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6"
>>>>>>> /usr/local/hbase/logs/hbase-hadoop-master-dev2.log.2008-08-29
>>>>>>> 2008-08-29 07:58:59,660 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to
>>>>
>>>>
>>>>> server 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,728 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,881 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,881 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 open on
>>>>>>> 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,881 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014953935 and server 10.131.237.54:60020
>>>>>>> M1 2008-08-29 08:50:18,558 INFO
>>>>>>> org.apache.hadoop.hbase.master.RegionManager: assigning region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server
>>>>>>> 10.131.237.51:60020
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> What happened in here to cause the master reassign a region already
>>>>>> assigned?  It was on 54 then we are asked assign it to 51.  Would
>>>>>> suggest you run with DEBUG.  Would help with the diagnosis (See FAQ
>>>>>> for
>>>>>> how).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 2008-08-29 08:50:21,935 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:50:25,086 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> ...
>>>>>>> 2008-08-29 08:53:06,020 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:53:06,020 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> M2 2008-08-29 08:53:36,875 INFO
>>>>>>> org.apache.hadoop.hbase.master.ServerManager: Received
>>>>>>> MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> The multiple instances of MSG_REPORT_PROCESS_OPEN would imply it was
>>>>>> taking the redeployment a long time to come up replaying edits (If its
>>>>>> replaying edits of a region already deployed, that'll make for a real
>>>>>>
>>>>>>
>>>>> mess).
>>>>
>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>> 2008-08-29 08:53:36,876 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 open on
>>>>>>> 10.131.237.51:60020
>>>>>>> M3 2008-08-29 08:53:36,876 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014954000 and server 10.131.237.51:60020
>>>>>>> 2008-08-29 08:54:12,249 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> Skipping region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because it is already closing.
>>>>>>> 2008-08-29 08:54:15,525 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> Skipping region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because it is already closing.
>>>>>>> 2008-08-29 08:54:18,581 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> Skipping region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because it is already closing.
>>>>>>> M4 2008-08-29 08:54:24,616 INFO
>>>>>>> org.apache.hadoop.hbase.master.ServerManager: Received
>>>>>>>
>>>>>>>
>>>>>> MSG_REPORT_CLOSE:
>>>>
>>>>
>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> M5 2008-08-29 08:54:24,937 INFO
>>>>>>> org.apache.hadoop.hbase.master.RegionManager: assigning region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to server
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:28,133 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.55:60020
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Now we are assigning the region to 55.  I don't know why the
>>>>>> MSG_REPORT_CLOSE happened over on 51.  Doesn't say in the regionserver
>>>>>> log.    Our logging needs improving.  Without DEBUG its not possible
>>>>>> to
>>>>>> know why some of these actions happened.
>>>>>>
>>>>>> St.Ack
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 2008-08-29 08:54:28,134 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:28,221 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 open on
>>>>>>> 10.131.237.55:60020
>>>>>>> M6 2008-08-29 08:54:28,221 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014319248 and server 10.131.237.55:60020
>>>>>>> M7 2008-08-29 08:54:30,775 INFO
>>>>>>> org.apache.hadoop.hbase.master.ServerManager: Received
>>>>>>>
>>>>>>>
>>>>>> MSG_REPORT_SPLIT:
>>>>
>>>>
>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774:
>>>>>>> [B@4861ef74from
>>>>>>> 10.131.237.51:60020
>>>>>>> M8 2008-08-29 08:54:33,356 INFO
>>>>>>> org.apache.hadoop.hbase.master.RegionManager: assigning region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to server
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:54:36,419 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:54:39,498 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> ...
>>>>>>> 2008-08-29 08:55:03,727 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> M9 2008-08-29 08:55:04,835 INFO
>>>>>>> org.apache.hadoop.hbase.master.ServerManager: Received
>>>>>>>
>>>>>>>
>>>>>> MSG_REPORT_SPLIT:
>>>>
>>>>
>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774:
>>>>>>> [B@62927d48from
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:55:06,898 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> ...
>>>>>>> 2008-08-29 08:56:04,627 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> M10 2008-08-29 08:56:04,627 INFO
>>>>>>> org.apache.hadoop.hbase.master.ServerManager: Received
>>>>>>> MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:04,642 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 open on
>>>>>>> 10.131.237.52:60020
>>>>>>> M11 2008-08-29 08:56:04,642 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014954468 and server 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:10,645 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:16,111 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to
>>>>
>>>>
>>>>> server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 open on
>>>>>>> 10.131.236.91:60020
>>>>>>> M12 2008-08-29 08:56:19,134 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014953772 and server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:43,935 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 to
>>>>
>>>>
>>>>> server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:47,158 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:50,274 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,421 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,422 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,500 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 open on
>>>>>>> 10.131.236.91:60020
>>>>>>> M13 2008-08-29 08:56:53,500 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014953772 and server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:57:02,485 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:57:04,188 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to
>>>>
>>>>
>>>>> server 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 open on
>>>>>>> 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220024985751 and server 10.131.237.53:60020
>>>>>>> 2008-08-29 09:00:45,252 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>
>>>>
>>>>> Deleting region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because daughter splits no longer hold references
>>>>>>> [hadoop@ice-dev2-buc ~]$ grep
>>>>>>>
>>>>>>>
>>>>>> "users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6"
>>>>
>>>>  /usr/local/hbase/logs/hbase-hadoop-master-ice-dev2-buc.corp.adobe.com.log.20
>>>>
>>>>
>>>>> 08-08-29
>>>>>>> 2008-08-29 07:58:59,660 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to
>>>>
>>>>
>>>>> server 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,728 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,881 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,881 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 open on
>>>>>>> 10.131.237.54:60020
>>>>>>> 2008-08-29 07:59:02,881 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014953935 and server 10.131.237.54:60020
>>>>>>> 2008-08-29 08:50:18,558 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to
>>>>
>>>>
>>>>> server 10.131.237.51:60020
>>>>>>> 2008-08-29 08:50:21,935 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:50:25,086 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> ...
>>>>>>> 2008-08-29 08:53:06,020 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:53:36,875 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:53:36,876 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 open on
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:53:36,876 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014954000 and server 10.131.237.51:60020
>>>>>>> 2008-08-29 08:54:12,249 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> Skipping region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because it is already closing.
>>>>>>> 2008-08-29 08:54:15,525 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> Skipping region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because it is already closing.
>>>>>>> 2008-08-29 08:54:18,581 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> Skipping region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because it is already closing.
>>>>>>> 2008-08-29 08:54:24,616 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:54:24,937 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 to
>>>>
>>>>
>>>>> server 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:28,133 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:28,134 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 from
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:28,221 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 open on
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:28,221 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014319248 and server 10.131.237.55:60020
>>>>>>> 2008-08-29 08:54:30,775 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_SPLIT:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774:
>>>>>>> [B@4861ef74from
>>>>>>> 10.131.237.51:60020
>>>>>>> 2008-08-29 08:54:33,356 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to
>>>>
>>>>
>>>>> server 10.131.237.52:60020
>>>>>>> 2008-08-29 08:54:36,419 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:54:39,498 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> ...
>>>>>>> 2008-08-29 08:55:03,727 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:55:04,835 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_SPLIT:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774:
>>>>>>> [B@62927d48from
>>>>>>> 10.131.237.55:60020
>>>>>>> 2008-08-29 08:55:06,898 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> ...
>>>>>>> 2008-08-29 08:56:04,627 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:04,627 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:04,642 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 open on
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:04,642 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014954468 and server 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:10,645 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.52:60020
>>>>>>> 2008-08-29 08:56:16,111 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to
>>>>
>>>>
>>>>> server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 open on
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:19,134 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014953772 and server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:43,935 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 to
>>>>
>>>>
>>>>> server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:47,158 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:50,274 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,421 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,422 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,500 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 open on
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:56:53,500 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220024667454 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220014953772 and server 10.131.236.91:60020
>>>>>>> 2008-08-29 08:57:02,485 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.236.91:60020
>>>>>>> 2008-08-29 08:57:04,188 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.RegionManager:
>>>>
>>>>
>>>>> assigning region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 to
>>>>
>>>>
>>>>> server 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_PROCESS_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.ServerManager:
>>>>
>>>>
>>>>> Received MSG_REPORT_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 from
>>>>>>> 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 open on
>>>>>>> 10.131.237.53:60020
>>>>>>> 2008-08-29 08:57:07,309 INFO
>>>>>>> org.apache.hadoop.hbase.master.ProcessRegionOpen$1: updating row
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231 in region
>>>>>>>
>>>>>>>
>>>>>> .META.,,1
>>>>
>>>>
>>>>> with startcode 1220024985751 and server 10.131.237.53:60020
>>>>>>> M14 2008-08-29 09:00:45,252 INFO
>>>>>>>
>>>>>>>
>>>>>> org.apache.hadoop.hbase.master.BaseScanner:
>>>>
>>>>
>>>>> Deleting region
>>>>>>>
>>>>>>>
>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>
>>>>
>>>>> because daughter splits no longer hold references
>>>>>>>
>>>>>>>
>>>>>>> Region server log 10.131.237.51
>>>>>>>
>>>>>>> grep users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6
>>>>>>> /usr/local/hbase/logs/hbase-hadoop-regionserver-dev1.log.2008-08-29
>>>>>>> 2008-08-29 08:50:18,868 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>>>> 2008-08-29 08:52:52,323 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>>>> R1 2008-08-29 08:53:34,330 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774/62578664
>>>>>>>
>>>>>>>
>>>>>> available
>>>>
>>>>
>>>>> R2 2008-08-29 08:53:34,330 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on
>>>>>>>
>>>>>>>
>>>>>> region
>>>>
>>>>
>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>>>> R3 2008-08-29 08:54:09,035 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@284f4565
>>>>>>> R4 2008-08-29 08:54:24,224 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on
>>>>>>>
>>>>>>>
>>>>>> region
>>>>
>>>>
>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 in 49sec
>>>>>>> R5 2008-08-29 08:54:24,224 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting split of
>>>>>>> region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>>>> R6 2008-08-29 08:54:24,234 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774
>>>>>>> R7 2008-08-29 08:54:24,393 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774: [B@284f4565
>>>>>>> R8 2008-08-29 08:54:24,393 WARN
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774 already
>>>>>>> closed
>>>>>>> R9 2008-08-29 08:54:27,323 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: region
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231/620189721
>>>>>>>
>>>>>>>
>>>>>> available
>>>>
>>>>
>>>>> R10 2008-08-29 08:54:27,324 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion: closed
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231
>>>>>>> R11 2008-08-29 08:54:27,726 INFO
>>>>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: region
>>>>>>> split,
>>>>>>>
>>>>>>>
>>>>>> META
>>>>
>>>>
>>>>> updated, and report to master all successful. Old region=REGION =>
>>>>>>>
>>>>>>>
>>>>>> {NAME =>
>>>>
>>>>
>>>>> 'users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220021933774', STARTKEY =>
>>>>>>> '27B5C924-6FE4-4484-B131-D5C9E71AF4A6', ENDKEY =>
>>>>>>> '2FB0249D-0A95-47BB-A4F1-6C7600039542', ENCODED => 62578664, OFFLINE
>>>>>>> =>
>>>>>>> true, SPLIT => true, TABLE => {{NAME => 'users', IS_ROOT => 'false',
>>>>>>>
>>>>>>>
>>>>>> IS_META
>>>>
>>>>
>>>>> => 'false', FAMILIES => [{NAME => 'test', BLOOMFILTER => 'false',
>>>>>>> COMPRESSION => 'NONE', VERSIONS => '3', LENGTH => '2147483647', TTL
>>>>>>> =>
>>>>>>>
>>>>>>>
>>>>>> '-1',
>>>>
>>>>
>>>>> IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}, new regions:
>>>>>>> users,27B5C924-6FE4-4484-B131-D5C9E71AF4A6,1220025264231,
>>>>>>> users,2BB6AC25-2EA6-4311-9670-49B68B7BB975,1220025264231. Split took
>>>>>>>
>>>>>>>
>>>>>> 3sec
>>>>
>>>>
>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>


Mime
View raw message