hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy
Date Fri, 02 Sep 2011 16:57:12 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096126#comment-13096126
] 

ramkrishna.s.vasudevan commented on HBASE-4015:
-----------------------------------------------

The following tests were performed

The cluster has 3 nodes.
Created some 4000+ regions.  And i followed the startegy like

Initially create regions

Balancer will balance the regions across all the three.

Using a script kill the RS in these 3 nodes at fixed time.(at different time intervals).

Then start it after sometime.

Sometimes once all the 3 RS are up invoke the balancer command manually.

While balancing kill one of the RS which was the destination during balancer calculation.
Again bring it up.

The number of regions remained constant and every time the Timeout detected it was able to
overcome it with no issues at all.

I also calculated some metrics

There were instance were 144, 249,1334, 30 regions were timed out.
For any timeout region to be monitored and completed it took at the max 1.5 sec minimum being
0.35 secs.

The following logs shows the scenario where the timeout monitor tries to assign a node to
OFFLINE but by the time the node is changed to OPENED by RS.
{code}
2011-09-02 18:15:55,621 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has
been PENDING_OPEN for too long, reassigning region=testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.
2011-09-02 18:15:55,709 INFO org.mortbay.log: Timeout detected for the region testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.
at Fri Sep 02 18:15:55 IST 2011
2011-09-02 18:15:55,942 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x13229c29f830010
Creating (or updating) unassigned node for b7a45cf31651d5cbba682ae24b83bf88 with OFFLINE state


2011-09-02 18:15:55,967 INFO org.apache.hadoop.hbase.zookeeper.ZKAssign: Current data in znode
.region=testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88., origin=HOST-10-18-52-198,60020,1314967317396,
state=RS_ZK_REGION_OPENED
2011-09-02 18:15:55,996 WARN org.apache.hadoop.hbase.master.AssignmentManager: Attempted to
create/force node into OFFLINE state before completing assignment but failed to do so for
testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88. state=PENDING_OPEN, ts=1314967555942,
server=null

 


2011-09-02 18:15:55,999 INFO org.mortbay.log: Time taken to complete the action on timeout
for the reigon testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.291
2011-09-02 18:16:24,115 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED,
server=HOST-10-18-52-198,60020,1314967317396, region=b7a45cf31651d5cbba682ae24b83bf88, which
is more than 15 seconds late
2011-09-02 18:16:24,122 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x13229c29f830010
Successfully deleted unassigned node for region b7a45cf31651d5cbba682ae24b83bf88 in expected
state RS_ZK_REGION_OPENED
{code}

> Refactor the TimeoutMonitor to make it less racy
> ------------------------------------------------
>
>                 Key: HBASE-4015
>                 URL: https://issues.apache.org/jira/browse/HBASE-4015
>             Project: HBase
>          Issue Type: Sub-task
>    Affects Versions: 0.90.3
>            Reporter: Jean-Daniel Cryans
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Blocker
>             Fix For: 0.92.0
>
>         Attachments: HBASE-4015_1_trunk.patch, HBASE-4015_2_trunk.patch, HBASE-4015_reprepared_trunk_2.patch,
Timeoutmonitor with state diagrams.pdf
>
>
> The current implementation of the TimeoutMonitor acts like a race condition generator,
mostly making things worse rather than better. It does it's own thing for a while without
caring for what's happening in the rest of the master.
> The first thing that needs to happen is that the regions should not be processed in one
big batch, because that sometimes can take minutes to process (meanwhile a region that timed
out opening might have opened, then what happens is it will be reassigned by the TimeoutMonitor
generating the never ending PENDING_OPEN situation).
> Those operations should also be done more atomically, although I'm not sure how to do
it in a scalable way in this case.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message