Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 29 Sep 2017 13:14:08 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12841977.1435753971000.236079.1506690848742@Atlassian.JIRA>
In-Reply-To: <JIRA.12841977.1435753971000@Atlassian.JIRA>
References: <JIRA.12841977.1435753971000@Atlassian.JIRA> <JIRA.12841977.1435753971889@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-14004) [Replication] Inconsistency
 between Memstore and WAL may result in data in remote cluster that is not
 in the origin
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 29 Sep 2017 13:14:14 -0000


    [ https://issues.apache.org/jira/browse/HBASE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185802#comment-16185802 ] 

Hudson commented on HBASE-14004:
--------------------------------

FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3798 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3798/])
HBASE-18845 TestReplicationSmallTests fails after HBASE-14004 (zhangduo: rev 239e6872674ff122ecec2d8d6a557b269e6ae54b)
* (edit) hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java


> [Replication] Inconsistency between Memstore and WAL may result in data in remote cluster that is not in the origin
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-14004
>                 URL: https://issues.apache.org/jira/browse/HBASE-14004
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver, Replication
>            Reporter: He Liangliang
>            Assignee: Duo Zhang
>            Priority: Critical
>              Labels: replication, wal
>             Fix For: 2.0.0-alpha-4
>
>         Attachments: HBASE-14004.patch, HBASE-14004-v1.patch, HBASE-14004-v2.patch, HBASE-14004-v2.patch, HBASE-14004-v3.patch
>
>
> Looks like the current write path can cause inconsistency between memstore/hfile and WAL which cause the slave cluster has more data than the master cluster.
> The simplified write path looks like:
> 1. insert record into Memstore
> 2. write record to WAL
> 3. sync WAL
> 4. rollback Memstore if 3 fails
> It's possible that the HDFS sync RPC call fails, but the data is already  (may partially) transported to the DNs which finally get persisted. As a result, the handler will rollback the Memstore and the later flushed HFile will also skip this record.
> ==================================
> This is a long lived issue. The above problem is solved by write path reorder, as now we will sync wal first before modifying memstore. But the problem may still exists as replication thread may read the new data before we return from hflush. See this document for more details:
> https://docs.google.com/document/d/11AyWtGhItQs6vsLRIx32PwTxmBY3libXwGXI25obVEY/edit#
> So we need to keep a sync length in WAL and tell replication wal reader this is limit when you read this wal file.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)