hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Yuan Jiang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-11099) Two situations where we could open a region with smaller sequence number
Date Mon, 17 Nov 2014 23:33:35 GMT

     [ https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Stephen Yuan Jiang updated HBASE-11099:
---------------------------------------
    Attachment: HBASE-11099.v1-2.0.patch

This patch fix the issue in HRegion#replayRecoveredEdits().  

Problem: 
HRegion#replayRecoveredEdits() returns the 'currentEditSeqId'.  In some situation that coprocessorHost
could skip some log entries.  If those log entries are at the tail of the logs, the returned
'currentEditSeqId' would not be updated (due to the code would skip the update).  This might
cause data loss due to "region open with a smaller sequence number and Master may record a
larger flushed sequence Id.  If the region fails again, some WALEdits maybe skipped during
recovery."

Fix:
Move the currentEditSeqId update ahead of the skip code.  

I also optimized a code to determine whether flush the log: once flush boolean is set to true,
we should not keep checking whether to flush (because we should flush).

> Two situations where we could open a region with smaller sequence number
> ------------------------------------------------------------------------
>
>                 Key: HBASE-11099
>                 URL: https://issues.apache.org/jira/browse/HBASE-11099
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.99.0
>            Reporter: Jeffrey Zhong
>            Assignee: Stephen Yuan Jiang
>             Fix For: 0.99.2
>
>         Attachments: HBASE-11099.v1-2.0.patch
>
>
> Recently I happened to run into code where we potentially could open region with smaller
sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the way WAL
Sync where we use late binding(assign sequence number right before wal sync).
> The flushSeqId may less than the change sequence number included in the flush which may
cause later region opening code to use a smaller than expected sequence number when we reopen
the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
>           if (coprocessorHost != null) {
>             status.setStatus("Running pre-WAL-restore hook in coprocessors");
>             if (coprocessorHost.preWALRestore(this.getRegionInfo(), key, val)) {
>               // if bypass this log entry, ignore it ...
>               continue;
>             }
>           }
> ...
>           currentEditSeqId = key.getLogSeqNum();
> {code} 
> If coprocessor skip some tail WALEdits, then the function will return smaller currentEditSeqId.
In the end, a region may also open with a smaller sequence number. This may cause data loss
because Master may record a larger flushed sequence Id and some WALEdits maybe skipped during
recovery if the region fail again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message