Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7E6B718254 for ; Tue, 23 Jun 2015 18:22:03 +0000 (UTC) Received: (qmail 68550 invoked by uid 500); 23 Jun 2015 18:22:03 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 68511 invoked by uid 500); 23 Jun 2015 18:22:03 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 68499 invoked by uid 99); 23 Jun 2015 18:22:03 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jun 2015 18:22:03 +0000 Date: Tue, 23 Jun 2015 18:22:03 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-13938) Deletes done during the region merge transaction may get eclipsed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598118#comment-14598118 ] Hudson commented on HBASE-13938: -------------------------------- FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #988 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/988/]) HBASE-13938 Deletes done during the region merge transaction may get eclipsed (ddas, enis, ndimiduk) (ndimiduk: rev 605fd4f33f6284ee818359911ddbea77111efb27) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeRequest.java * hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-protocol/src/main/protobuf/Admin.proto * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/VisibilityLabelsProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java > Deletes done during the region merge transaction may get eclipsed > ----------------------------------------------------------------- > > Key: HBASE-13938 > URL: https://issues.apache.org/jira/browse/HBASE-13938 > Project: HBase > Issue Type: Bug > Components: master, regionserver > Reporter: Devaraj Das > Assignee: Enis Soztutar > Fix For: 0.98.14, 1.0.2, 1.2.0, 1.1.1, 1.3.0 > > Attachments: 13938-branch-1.1.txt, hbase-13938_master.patch, hbase-13938_v2-branch-1.1.patch, hbase-13938_v3-0.98.patch, hbase-13938_v3-0.98.patch, hbase-13938_v3-branch-1.0.patch, hbase-13938_v3-branch-1.1.patch, hbase-13938_v3-branch-1.2.patch, hbase-13938_v3-branch-1.patch > > > Was looking at an issue from our internal testing. It seems the Deletes of the region rows from the meta done during the merge transaction could be eclipsed by the Put of a region row that might have happened moments before. > The master logs this for the merge: > {noformat} > 2015-06-18 13:13:46,018 INFO [AM.ZK.Worker-pool2-t12] master.AssignmentManager: Handled MERGED event; merged=IntegrationTestIngest,a666665c,1434633226681.0927319db6bf5e128e3bec2a420819aa., region_a=IntegrationTestIngest,a666665c,1434631353820.8b911862d7705ac808b8d132d0154c16., region_b=IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec., on ddas-2-5.openstacklocal,16020,1434632778438 > {noformat} > One of the regions that got merged got Opened a few seconds back: > {noformat} > 2015-06-18 13:13:46,591 INFO [RS_OPEN_REGION-ddas-2-5:16020-1] regionserver.HRegion: Onlined 1bdaf759862f45d133ef77fdbda21aec; next sequenceid=182988 > {noformat} > The above would have done a Put in the meta. > Looking at the raw scan of the meta, for the new merged region, the creation timestamp is 1434633226101: > {noformat} > IntegrationTestIngest,a666665c,1434633226681.0927319db6bf5e128e3bec2a420819aa. column=info:regioninfo, timestamp=1434633226101, value={ENCODED => 0927319db6bf5e128e3bec2a420819aa, NAME => 'IntegrationTestIngest,a666665c,1434633226681.0927319db6bf5e128e3bec2a420819aa.', STARTKEY => 'a666665c', ENDKEY => 'b3333328'} > {noformat} > Looking at the raw scan of the meta, the timestamp for the region open of the already merged region is 1434633226600. This is a little after the merge transaction's timestamp. > {noformat} > IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec. column=info:seqnumDuringOpen, timestamp=1434633226600, value=\x00\x00\x00\x00\x00\x02\xCA\xCC > IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec. column=info:server, timestamp=1434633226600, value=ddas-2-5.openstacklocal:16020 > IntegrationTestIngest,acccccc2,1434631353820.1bdaf759862f45d133ef77fdbda21aec. column=info:serverstartcode, timestamp=1434633226600, value=1434632778438 > {noformat} > We need to fix it so that the merge region transaction also takes the master's timestamp. Similar to HBASE-13875. > When this happens, clients start to see a row in the meta with an empty HRegionInfo (this is because the Put done during the region open only updates the location information but not the HRI, and the HRI deleted during the merge transaction "remains deleted"). -- This message was sent by Atlassian JIRA (v6.3.4#6332)