hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chunhui shen (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-4899) Region would be assigned twice easily with continually killing server and moving region in testing environment
Date Wed, 30 Nov 2011 06:15:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159851#comment-13159851
] 

chunhui shen commented on HBASE-4899:
-------------------------------------

{code}
Oct 20 12:51:36 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,345 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting c70b69079782d421b34ac5e57ef06a35 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:37 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,472 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a07d62ba400f631c781d7232ee397ed3 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:39 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,756 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 05fb66d83cf2e4294a3cbd3f6757ec32 on serverName=dw90.kgb.sqa.cm4,60020,1319083018625,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:40 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,805 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting e53b5cb8fcaaa178270f2dc876ebdd9a on serverName=dw90.kgb.sqa.cm4,60020,1319083018625,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:43 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,825 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting d9ae4de5ba9045f4b060191222f774fd on serverName=dw81.kgb.sqa.cm4,60020,1319083018636,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,889 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 589e9269c31cdcd8aa073518c133f286 on serverName=dw90.kgb.sqa.cm4,60020,1319083018625,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,892 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting f35352a47f076c33a0f5ea140f76c943 on serverName=dw90.kgb.sqa.cm4,60020,1319083018625,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,964 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 5fecc8e3a17eed75ba7bb2de2bb5c762 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,009 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 0cdf089748ecdc79123670bf76dc68d5 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,071 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 6245b4e8b567755509ba6351ded55c92 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,380 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting fd52c86737233950bf509fb9ecf524a9 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,461 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a30b7035359910a63d34b99402a9ef35 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:45 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,918 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 363e647e5b9c96c2ad1acc6e44471b74 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,130 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting e996ca8ca0eede508af4e02964ae3de6 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,155 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting fb9fc31c5baa604fb76afd9df419ca8b on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,155 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting fb9fc31c5baa604fb76afd9df419ca8b on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,320 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting e9cbaa732a070c8947eb0603a84634c5 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,417 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a68d33b7bbbb4bfecee9679e8af6b991 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,494 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a708d26df7e2c62d09851eb0053cb567 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,513 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting b8bc0ba5ef9fd610b38afd76891b5506 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,835 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting adf8c774655437f14a72552eed6566c8 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 12:51:48 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,984 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 555048e18d23e8d88ce34f3b2798d893 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:26 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,731 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 4ff6a541f09be443db27482ac170077b on serverName=dw83.kgb.sqa.cm4,60020,1319087456454,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:26 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,757 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 9203d1b6f17eaec9f62d3252cc0a4728 on serverName=dw90.kgb.sqa.cm4,60020,1319087454876,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:28 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,963 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 05bd751ae0894c2618b431ee03e33536 on serverName=dw79.kgb.sqa.cm4,60020,1319087456067,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:28 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,989 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 9d619820bc07ffbc6260fe8cac3ac8fb on serverName=dw79.kgb.sqa.cm4,60020,1319087456067,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:28 dw75.kgb.sqa.cm4 2011-10-20 13:12:01,192 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting e53b5cb8fcaaa178270f2dc876ebdd9a on serverName=dw79.kgb.sqa.cm4,60020,1319087456067,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:33 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,401 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting aeb0ed71b9866626a79313d6f43e04d2 on serverName=dw90.kgb.sqa.cm4,60020,1319087454876,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:34 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,487 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting c2e4e403709ab6c41d9a2d9bc74e758e on serverName=dw90.kgb.sqa.cm4,60020,1319087454876,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:34 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,614 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 10fc5766fd94fd1e306c2f1b73e1476f on serverName=dw90.kgb.sqa.cm4,60020,1319087454876,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:34 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,614 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 10fc5766fd94fd1e306c2f1b73e1476f on serverName=dw90.kgb.sqa.cm4,60020,1319087454876,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:37 dw75.kgb.sqa.cm4 2011-10-20 13:12:03,836 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 3e5da14300d6af039221c29564c5f3e3 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:37 dw75.kgb.sqa.cm4 2011-10-20 13:12:03,942 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a68d33b7bbbb4bfecee9679e8af6b991 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:37 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,240 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 2d26053bb10898a5f203ce96e93a35b1 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:38 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,271 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a708d26df7e2c62d09851eb0053cb567 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:38 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,292 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting e996ca8ca0eede508af4e02964ae3de6 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:39 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,463 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 901c30cb291925739451ef6997cd9750 on serverName=dw83.kgb.sqa.cm4,60020,1319087456454,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:39 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,794 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting f39019fda9968274f0d6b44cc9281004 on serverName=dw79.kgb.sqa.cm4,60020,1319087456067,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,019 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting b140350ff7de2b2d2fcd353ef1b565d2 on serverName=dw90.kgb.sqa.cm4,60020,1319087454876,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,089 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 47b40d219b1dba4b210b00ac7de13a6c on serverName=dw83.kgb.sqa.cm4,60020,1319087456454,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,306 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 47893f2ac30964f06f8f65219077b60f on serverName=dw79.kgb.sqa.cm4,60020,1319087456067,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,351 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting a38ef110e6ef3a06f1fa1c25fba5be23 on serverName=dw79.kgb.sqa.cm4,60020,1319087456067,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)
Oct 20 17:11:17 dw75.kgb.sqa.cm4 2011-10-20 17:11:17,048 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 70236052 on serverName=dw79.kgb.sqa.cm4,60020,1319101778268, load=(requests=0,
regions=0, usedHeap=0, maxHeap=0)
Oct 20 21:35:24 dw75.kgb.sqa.cm4 2011-10-20 21:35:24,312 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Overwriting 88bd3816503151926f30916066750e08 on serverName=dw79.kgb.sqa.cm4,60020,1319114721135,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0)

{code}
                
> Region would be assigned twice easily with continually  killing server and moving region
in testing environment
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-4899
>                 URL: https://issues.apache.org/jira/browse/HBASE-4899
>             Project: HBase
>          Issue Type: Bug
>            Reporter: chunhui shen
>
> Before assigning region in ServerShutdownHandler#process, it will check whether region
is in RIT,
> however, this checking doesn't work as the excepted in the following case:
> 1.move region A from server B to server C
> 2.kill server B
> 3.start server B immediately
> Let's see what happen in the code for the above case
> {code}
> for step1:
> 1.1 server B close the region A,
> 1.2 master setOffline for region A,(AssignmentManager#setOffline:this.regions.remove(regionInfo))
> 1.3 server C start to open region A.(Not completed)
> for step3:
> master ServerShutdownHandler#process() for server B
> {
> ..
> splitlog()
> ...
> List<RegionState> regionsInTransition =
>         this.services.getAssignmentManager()
>         .processServerShutdown(this.serverName);
> ...
> Skip regions that were in transition unless CLOSING or PENDING_CLOSE
> ...
> assign region
> }
> In fact, when running ServerShutdownHandler#process()#this.services.getAssignmentManager().processServerShutdown(this.serverName),
region A is in RIT (step1.3 not completed), but the return List<RegionState> regionsInTransition
doesn't contain it, because region A has removed from AssignmentManager.regions by AssignmentManager#setOffline
in step 1.2
> Therefore, region A will be assigned twice.
> {code}
> Actually, one server killed and started twice will also easily cause region assigned
twice.
> Exclude the above reason, another probability : 
> when execute ServerShutdownHandler#process()#MetaReader.getServerUserRegions ,region
is included which is in RIT now.
> But after completing MetaReader.getServerUserRegions, the region has been opened in other
server and is not in RIT now.
> In our testing environment where balancing,moving and killing are executed periodly,
assigning region twice often happens, and it is hateful because it will affect other test
cases.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message