Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3AF6F1775B for ; Fri, 31 Oct 2014 20:34:35 +0000 (UTC) Received: (qmail 20250 invoked by uid 500); 31 Oct 2014 20:34:35 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 20193 invoked by uid 500); 31 Oct 2014 20:34:35 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 20178 invoked by uid 99); 31 Oct 2014 20:34:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2014 20:34:34 +0000 Date: Fri, 31 Oct 2014 20:34:34 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around the same timeframe MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192432#comment-14192432 ] Hudson commented on HBASE-12377: -------------------------------- ABORTED: Integrated in HBase-1.0 #401 (See [https://builds.apache.org/job/HBase-1.0/401/]) HBASE-12377 HBaseAdmin#deleteTable fails when META region is moved around the same timeframe (Stephen Yuan Jiang) (enis: rev 8938aba4e265dd4d94515312df2a9ebe63ab544f) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java > HBaseAdmin#deleteTable fails when META region is moved around the same timeframe > -------------------------------------------------------------------------------- > > Key: HBASE-12377 > URL: https://issues.apache.org/jira/browse/HBASE-12377 > Project: HBase > Issue Type: Bug > Components: Client > Affects Versions: 0.98.4 > Reporter: Stephen Yuan Jiang > Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12377.v1-2.0.patch, HBASE-12377.v2-2.0.patch, HBASE-12377.v3-2.0.patch > > > This is the same issue that HBASE-10809 tried to address. The fix of HBASE-10809 refetch the latest meta location in retry-loop. However, there are 2 problems: (1). inside the retry loop, there is another try-catch block that would throw the exception before retry can kick in; (2). It looks like that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data from meta cache, which means if the meta cache is stale and out of date, retries would not solve the problem by fetching from the stale meta cache. > Here is the call stack of the issue: > {noformat} > 2014-10-27 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-172-31-0-48.ec2.internal,60020,1414403435009 > 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2774) > 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4257) > 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3156) > 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994) > 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) > 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) > 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) > 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) > 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.Thread.run(Thread.java:745) > 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread| > 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source) > 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:306) > 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:699) > 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:654) > 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.IntegrationTestManyRegions.tearDown(IntegrationTestManyRegions.java:99) > {noformat} > The META region was Online in RS1 when the delete table starts, it was moved to RS2 during the delete table operation. And the problem appears. -- This message was sent by Atlassian JIRA (v6.3.4#6332)