hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17407) Correct update of maxFlushedSeqId in HRegion
Date Thu, 05 Jan 2017 14:40:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15801513#comment-15801513

stack commented on HBASE-17407:

bq. (but for some reason the comment was deleted).

I intentionally deleted the comment because I felt it added little benefit to the back-and-forth

bq. I think it was important to understand that in the current state there is no danger of
data loss.

You mean with the finalizeFlush/updateStore calls in place and NO inmemory compaction -- just
BASIC mode where we flush all in the pipeline?

If the above, I think so. That said, the finalizeFlush/updateStore calls are new moving pieces
and this corner cases are hard to manufacture.

bq. Code maintainability is also important.

Yes. This sequenceid accounting is unfortunately involved and tough to test.

bq. I can replace finalizeFlush with a preFlushSeqIDEstimation() which returns a lower bound
on the sequence id that is invoked before we start the flush. 

You think this will restore our sequence id accounting to what it was before finalizeFlush/updateStore
?  How will we deal with the gap between the new edits coming in filling lowestUnflushedSequenceIds
 after we have swapped it out to do the current and the edits in the pipeline that did not
get flushed during the current flush session?

bq. You say WAL truncation cannot be triggered during a flush. 

Indeed. See how closeBarrier is used in AbstractFSWAL

bq. Can the map in seq accounting be reported to master during a flush?

See HRegion#setCompleteSequenceId where we build our sequenceid to send to the master.  See
how it asks the WAL subsystem for earliest edit by column family:

      long earliest = this.wal.getEarliestMemstoreSeqNum(encodedRegionName, familyName);

Here is the implementation:

  public long getEarliestMemstoreSeqNum(byte[] encodedRegionName, byte[] familyName) {
    // This method is used by tests and for figuring if we should flush or not because our
    // sequenceids are too old. It is also used reporting the master our oldest sequenceid
for use
    // figuring what edits can be skipped during log recovery. getEarliestMemStoreSequenceId
    // from this.sequenceIdAccounting is looking first in flushingOldestStoreSequenceIds,
    // currently flushing sequence ids, and if anything found there, it is returning these.
This is
    // the right thing to do for the reporting oldest sequenceids to master; we won't skip
edits if
    // we crash during the flush. For figuring what to flush, we might get requeued if our
    // id is old even though we are currently flushing. This may mean we do too much flushing.
    return this.sequenceIdAccounting.getLowestSequenceId(encodedRegionName, familyName);

It tries to explain how it works.

That it returns flushingSequenceIds and then lowestUnflushedSequenceIds if former is not present
may be what [~Apache9] is referring to in the 'not report the value if a flush ongoing' (I
did not see a block on reporting during 'flush' -- maybe I'm looking in wrong place).


> Correct update of maxFlushedSeqId in HRegion
> --------------------------------------------
>                 Key: HBASE-17407
>                 URL: https://issues.apache.org/jira/browse/HBASE-17407
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Eshcar Hillel
> The attribute maxFlushedSeqId in HRegion is used to track the max sequence id in the
store files and is reported to HMaster. When flushing only part of the memstore content this
value might be incorrect and may cause data loss.

This message was sent by Atlassian JIRA

View raw message