hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prakash Khemani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-3845) data loss because lastSeqWritten can miss memstore edits
Date Mon, 25 Jul 2011 14:51:10 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070542#comment-13070542
] 

Prakash Khemani commented on HBASE-3845:
----------------------------------------

In the patch that is deployed internally we have implemented a different approach. We remove
the region's entry in startCacheFlush() and save it (as opposed to the current behavior of
removing the entry in completeCacheFlush()). If the flush aborts then we restore the saved
entry.

The approach taken in the latest patch in this jira might also be OK. I have a few comments

{noformat}
           this.lastSeqWritten.remove(encodedRegionName);
+          Long seqWhileFlush = this.seqWrittenWhileFlush.get(encodedRegionName);
+          if (null != seqWhileFlush) {
+            this.lastSeqWritten.putIfAbsent(encodedRegionName, seqWhileFlush);
+            this.seqWrittenWhileFlush.remove(encodedRegionName);
+   
{noformat}

seqWrittenWhileFlush .get() and subsequent .remove() can be replaced by a single .remove()
{code}
Long seqWhileFlush = this.seqWrittenWhileFlush.remove(encodedRegionName);
if (null != seqWhileFlush) {
  lSW.put(encodedRegionName, seqWhileFlush);
else
  lSW.remove(encodedRegionName);
{code}

==
The bigger problem here is that completeCacheFlush() is not called with updatedLock acquired.
Therefore there might still be correctness issues with the latest patch.

==

{noformat}
   public void abortCacheFlush() {
+    this.isFlushInProgress.set(false);
     this.cacheFlushLock.unlock();
   }
{noformat}
shouldn't seqWrittenWhileFlush be cleaned up in abort cache?


> data loss because lastSeqWritten can miss memstore edits
> --------------------------------------------------------
>
>                 Key: HBASE-3845
>                 URL: https://issues.apache.org/jira/browse/HBASE-3845
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.90.3
>            Reporter: Prakash Khemani
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Critical
>             Fix For: 0.90.5
>
>         Attachments: HBASE-3845_1.patch, HBASE-3845_2.patch, HBASE-3845_4.patch, HBASE-3845_5.patch,
HBASE-3845__trunk.patch, HBASE-3845_trunk_2.patch
>
>
> (I don't have a test case to prove this yet but I have run it by Dhruba and Kannan internally
and wanted to put this up for some feedback.)
> In this discussion let us assume that the region has only one column family. That way
I can use region/memstore interchangeably.
> After a memstore flush it is possible for lastSeqWritten to have a log-sequence-id for
a region that is not the earliest log-sequence-id for that region's memstore.
> HLog.append() does a putIfAbsent into lastSequenceWritten. This is to ensure that we
only keep track  of the earliest log-sequence-number that is present in the memstore.
> Every time the memstore is flushed we remove the region's entry in lastSequenceWritten
and wait for the next append to populate this entry again. This is where the problem happens.
> step 1:
> flusher.prepare() snapshots the memstore under HRegion.updatesLock.writeLock().
> step 2 :
> as soon as the updatesLock.writeLock() is released new entries will be added into the
memstore.
> step 3 :
> wal.completeCacheFlush() is called. This method removes the region's entry from lastSeqWritten.
> step 4:
> the next append will create a new entry for the region in lastSeqWritten(). But this
will be the log seq id of the current append. All the edits that were added in step 2 are
missing.
> ==
> as a temporary measure, instead of removing the region's entry in step 3 I will replace
it with the log-seq-id of the region-flush-event.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message