Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 3 Sep 2014 04:17:51 +0000 (UTC)
From: "Andrew Purtell (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12738177.1409539665000.3157.1409717871982@Atlassian.JIRA>
In-Reply-To: <JIRA.12738177.1409539665000@Atlassian.JIRA>
References: <JIRA.12738177.1409539665000@Atlassian.JIRA>
 <JIRA.12738177.1409539665618@arcas>
Subject: [jira] [Resolved] (HBASE-11868) Data loss in hlog when the hdfs is
 unavailable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HBASE-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell resolved HBASE-11868.
------------------------------------
       Resolution: Fixed
    Fix Version/s:     (was: 0.98.7)
                   0.98.6

Applied v2 patch and pushed to 0.98. Tests pass locally.

> Data loss in hlog when the hdfs is unavailable
> ----------------------------------------------
>
>                 Key: HBASE-11868
>                 URL: https://issues.apache.org/jira/browse/HBASE-11868
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.5
>            Reporter: Liu Shaohui
>            Assignee: Liu Shaohui
>            Priority: Blocker
>             Fix For: 0.98.6
>
>         Attachments: HBASE-11868-0.98-v1.diff, HBASE-11868-0.98-v2.diff
>
>
> When using the new thread model in hbase 0.98, we found a bug which may cause data loss when the the hdfs is unavailable.
> When writing wal Edits to hlog in doMiniBatchMutation of HRegion, the hlog first call appendNoSync to write the edits to hlog and then call sync with txid. 
> Assumed that the txid of current write is 10, and the syncedTillHere in hlog is 9 and the failedTxid is 0. When  the the hdfs is unavailable, the AsyncWriter or AsyncSyncer will fail to apend the edits or sync, then they will update the syncedTillHere to 10 and the failedTxid to 10.
> When the hlog calls the sync with txid :10, the failedTxid will nerver be checked for txid equals with syncedTillHere.  The client thinks the write success , but the data only be writtten to memstore not hlog. If the regionserver is down later before the memstore is flushed, the data will be lost.
> See: FSHLog.java #1348
> {code}
>   // sync all transactions upto the specified txid
>   private void syncer(long txid) throws IOException {
>     synchronized (this.syncedTillHere) {
>       while (this.syncedTillHere.get() < txid) {
>         try {
>           this.syncedTillHere.wait();
>           if (txid <= this.failedTxid.get()) {
>             assert asyncIOE != null :
>               "current txid is among(under) failed txids, but asyncIOE is null!";
>             throw asyncIOE;
>           }
>         } catch (InterruptedException e) {
>           LOG.debug("interrupted while waiting for notification from AsyncNotifier");
>         }
>       }
>     }
>   }
> {code}
> We can fix this issue by moving the comparing of txid and failedTxid outside the while block.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)