hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9773) Master aborted when hbck asked the master to assign a region that was already online
Date Wed, 16 Oct 2013 00:37:42 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796254#comment-13796254
] 

Hadoop QA commented on HBASE-9773:
----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12608601/trunk-9773_v2.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 3 new or modified
tests.

    {color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 1.0 profile.

    {color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 2.0 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

    {color:red}-1 site{color}.  The patch appears to cause mvn site goal to fail.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7557//console

This message is automatically generated.

> Master aborted when hbck asked the master to assign a region that was already online
> ------------------------------------------------------------------------------------
>
>                 Key: HBASE-9773
>                 URL: https://issues.apache.org/jira/browse/HBASE-9773
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Jimmy Xiang
>         Attachments: trunk-9773.patch, trunk-9773_v2.patch
>
>
> Came across this situation (with a version of 0.96 very close to RC5 version created
on 10/11):
> The sequence of events that happened:
> 1. The hbck tool couldn't communicate with the RegionServer hosting namespace region
due to some security exceptions. hbck INCORRECTLY assumed the region was not deployed.
> In output.log (client side):
> {noformat}
> 2013-10-12 10:42:57,067|beaver.machine|INFO|ERROR: Region { meta => hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
hdfs => hdfs://gs-hdp2-secure-1381559462-hbase-12.cs1cloud.internal:8020/apps/hbase/data/data/hbase/namespace/a0ac0825ba2d0830614e7f808f31787a,
deployed =>  } not deployed on any region server.
> 2013-10-12 10:42:57,067|beaver.machine|INFO|Trying to fix unassigned region...
> {noformat}
> 2. This led to the hbck tool trying to tell the master to "assign" the region.
> In master log (hbase-hbase-master-gs-hdp2-secure-1381559462-hbase-12.log):
> {noformat}
> 2013-10-12 10:52:35,960 INFO  [RpcServer.handler=4,port=60000] master.HMaster: Client=hbase//172.18.145.105
assign hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 3. The master went through the steps - sent a CLOSE to the RegionServer hosting namespace
region.
> From master log:
> {noformat}
> 2013-10-12 10:52:35,981 DEBUG [RpcServer.handler=4,port=60000] master.AssignmentManager:
Sent CLOSE to gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794 for
region hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.
> {noformat}
> 4. The master then tried to assign the namespace region to a region server, and in the
process ABORTED:
> From master log:
> {noformat}
> 2013-10-12 10:52:36,025 DEBUG [RpcServer.handler=4,port=60000] master.AssignmentManager:
No previous transition plan found (or ignoring an existing plan) for hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.;
generated random plan=hri=hbase:namespace,,1381564449706.a0ac0825ba2d0830614e7f808f31787a.,
src=, dest=gs-hdp2-secure-1381559462-hbase-9.cs1cloud.internal,60020,1381564439807; 4 (online=4,
available=4) available servers, forceNewPlan=true
> 2013-10-12 10:52:36,026 FATAL [RpcServer.handler=4,port=60000] master.HMaster: Master
server abort: loaded coprocessors are: [org.apache.hadoop.hbase.security.access.AccessController]
> 2013-10-12 10:52:36,027 FATAL [RpcServer.handler=4,port=60000] master.HMaster: Unexpected
state : {a0ac0825ba2d0830614e7f808f31787a state=OPEN, ts=1381564451344, server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
.. Cannot transit it to OFFLINE.
> java.lang.IllegalStateException: Unexpected state : {a0ac0825ba2d0830614e7f808f31787a
state=OPEN, ts=1381564451344, server=gs-hdp2-secure-1381559462-hbase-1.cs1cloud.internal,60020,1381564439794}
.. Cannot transit it to OFFLINE.
> {noformat}
> {code}AssignmentManager.assign(HRegionInfo region, boolean setOfflineInZK, boolean forceNewPlan){code}
is the method that does all the above. This was called from the HMaster with true for both
the boolean arguments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message