Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 277631021E for ; Sat, 10 Aug 2013 18:21:49 +0000 (UTC) Received: (qmail 49810 invoked by uid 500); 10 Aug 2013 18:21:48 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 49677 invoked by uid 500); 10 Aug 2013 18:21:48 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 49653 invoked by uid 99); 10 Aug 2013 18:21:48 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 10 Aug 2013 18:21:48 +0000 Date: Sat, 10 Aug 2013 18:21:48 +0000 (UTC) From: "stack (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-9188) TestHBaseFsck#testNotInMetaOrDeployedHole occasionally fails MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13735989#comment-13735989 ] stack commented on HBASE-9188: ------------------------------ You have a fix [~ted_yu]? > TestHBaseFsck#testNotInMetaOrDeployedHole occasionally fails > ------------------------------------------------------------ > > Key: HBASE-9188 > URL: https://issues.apache.org/jira/browse/HBASE-9188 > Project: HBase > Issue Type: Bug > Reporter: Ted Yu > > From https://builds.apache.org/job/hbase-0.95-on-hadoop2/231/testReport/org.apache.hadoop.hbase.util/TestHBaseFsck/testNotInMetaOrDeployedHole/ (region tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153): > expected:<[NOT_IN_META_OR_DEPLOYED, HOLE_IN_REGION_CHAIN]> but was:<[NOT_IN_META_OR_DEPLOYED, NOT_DEPLOYED, HOLE_IN_REGION_CHAIN]> > Here is snippet of test output: > {code} > 2013-08-10 11:53:16,941 DEBUG [RS_CLOSE_REGION-vesta:38578-1] handler.CloseRegionHandler(168): set region closed state in zk successfully for region tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153. sn name: vesta.apache.org,38578,1376135290018 > 2013-08-10 11:53:16,941 DEBUG [RS_CLOSE_REGION-vesta:38578-1] handler.CloseRegionHandler(177): Closed region tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153. > 2013-08-10 11:53:16,942 DEBUG [AM.ZK.Worker-pool-2-thread-13] master.AssignmentManager(782): Handling transition=RS_ZK_REGION_CLOSED, server=vesta.apache.org,38578,1376135290018, region=3ec6178a369a899c007fd89807b37153, current state from region state map ={3ec6178a369a899c007fd89807b37153 state=PENDING_CLOSE, ts=1376135596730, server=vesta.apache.org,38578,1376135290018} > 2013-08-10 11:53:16,942 WARN [AM.ZK.Worker-pool-2-thread-13] master.RegionStates(245): Closed region 3ec6178a369a899c007fd89807b37153 still on vesta.apache.org,38578,1376135290018? Ignored, reset it to null > 2013-08-10 11:53:16,942 INFO [AM.ZK.Worker-pool-2-thread-13] master.RegionStates(260): Transitioned from {3ec6178a369a899c007fd89807b37153 state=PENDING_CLOSE, ts=1376135596730, server=vesta.apache.org,38578,1376135290018} to {3ec6178a369a899c007fd89807b37153 state=CLOSED, ts=1376135596942, server=null} > 2013-08-10 11:53:16,942 DEBUG [AM.ZK.Worker-pool-2-thread-13] handler.ClosedRegionHandler(92): Handling CLOSED event for 3ec6178a369a899c007fd89807b37153 > 2013-08-10 11:53:16,942 DEBUG [AM.ZK.Worker-pool-2-thread-13] master.AssignmentManager(1462): Table being disabled so deleting ZK node and removing from regions in transition, skipping assignment of region tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153. > ... > 2013-08-10 11:53:17,319 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(1815): getMetaTableRows: row -> tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153.{ENCODED => 3ec6178a369a899c007fd89807b37153, NAME => 'tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153.', STARTKEY => 'B', ENDKEY => 'C'} > 2013-08-10 11:53:17,320 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(1815): getMetaTableRows: row -> tableNotInMetaOrDeployedHole,C,1376135595424.c2ae2bddbe9302c4344c13936248ac9d.{ENCODED => c2ae2bddbe9302c4344c13936248ac9d, NAME => 'tableNotInMetaOrDeployedHole,C,1376135595424.c2ae2bddbe9302c4344c13936248ac9d.', STARTKEY => 'C', ENDKEY => ''} > 2013-08-10 11:53:17,320 INFO [pool-1-thread-1] util.TestHBaseFsck(231): tableNotInMetaOrDeployedHole,,1376135595423.9df585f7f666e1cd55d7b875aae22ece. > 2013-08-10 11:53:17,320 INFO [pool-1-thread-1] util.TestHBaseFsck(231): tableNotInMetaOrDeployedHole,A,1376135595424.90a7d5f2211951d321c9f29f4059671f. > 2013-08-10 11:53:17,320 INFO [pool-1-thread-1] util.TestHBaseFsck(231): tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153. > 2013-08-10 11:53:17,320 INFO [pool-1-thread-1] util.TestHBaseFsck(231): tableNotInMetaOrDeployedHole,C,1376135595424.c2ae2bddbe9302c4344c13936248ac9d. > 2013-08-10 11:53:17,326 DEBUG [pool-1-thread-1] client.ClientScanner(218): Finished region={ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} > 2013-08-10 11:53:17,327 INFO [pool-1-thread-1] util.TestHBaseFsck(319): {ENCODED => 9df585f7f666e1cd55d7b875aae22ece, NAME => 'tableNotInMetaOrDeployedHole,,1376135595423.9df585f7f666e1cd55d7b875aae22ece.', STARTKEY => '', ENDKEY => 'A'}vesta.apache.org,41438,1376135289941 > 2013-08-10 11:53:17,328 INFO [pool-1-thread-1] util.TestHBaseFsck(319): {ENCODED => 90a7d5f2211951d321c9f29f4059671f, NAME => 'tableNotInMetaOrDeployedHole,A,1376135595424.90a7d5f2211951d321c9f29f4059671f.', STARTKEY => 'A', ENDKEY => 'B'}vesta.apache.org,38578,1376135290018 > 2013-08-10 11:53:17,328 INFO [pool-1-thread-1] util.TestHBaseFsck(283): RegionName: tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153. > 2013-08-10 11:53:17,328 INFO [pool-1-thread-1] util.TestHBaseFsck(287): Undeploying region {ENCODED => 3ec6178a369a899c007fd89807b37153, NAME => 'tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153.', STARTKEY => 'B', ENDKEY => 'C'} from server vesta.apache.org,38578,1376135290018 > 2013-08-10 11:53:17,328 INFO [RpcServer.handler=1,port=38578] regionserver.HRegionServer(3612): Received close region: 3ec6178a369a899c007fd89807b37153Transitioning in ZK: no. Version of ZK closing node:-1. Destination server:null > 2013-08-10 11:53:17,329 ERROR [RpcServer.handler=1,port=38578] regionserver.HRegionServer(2473): Received CLOSE for a region which is not online, and we're not opening. > 2013-08-10 11:53:17,330 WARN [pool-1-thread-1] util.HBaseFsckRepair(156): Exception when closing region: tableNotInMetaOrDeployedHole,B,1376135595424.3ec6178a369a899c007fd89807b37153. > org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: The region 3ec6178a369a899c007fd89807b37153 is not online, and is not opening. > at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2476) > at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:3617) > at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:14458) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2147) > at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1854) > {code} > Region was not deployed after hbck run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira