hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13811) Splitting WALs, we are filtering out too many edits -> DATALOSS
Date Tue, 02 Jun 2015 17:16:50 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569443#comment-14569443
] 

stack commented on HBASE-13811:
-------------------------------

This has taken me a while. Issue is accounting of per-store sequenceids added by the flush-by-column-family
feature. This is enabled in branch-1, which is what I am testing.

Issue happens if RS crashes when in the middle of a flush. What I am seeing is that on split
of the WALs subsequent to the crash, we are skipping in essence a whole hfile of edits --
the hfile that would have been written out by the crash-curtailed flush.

Happens, of course, only when a bit of scale involved... so chaos monkey has chance to bring
on the condition (happens about 1 in 3 runs).

After a couple of false trails, I now suspect this bit of code in FSHLog:

    private void updateOldestUnflushedSequenceIds(byte[] encodedRegionName,
        Set<byte[]> familyNameSet, Long lRegionSequenceId) {
      ConcurrentMap<byte[], Long> m =
        getOrCreateOldestUnflushedStoreSequenceIdsOfRegion(encodedRegionName);
      for (byte[] familyName : familyNameSet) {
        m.putIfAbsent(familyName, lRegionSequenceId);
      }
    }

When we go to flush, we REMOVE from the map of oldest outstanding sequenceids, all reference
to those we are about to flush -- we move them to a new map which is used when the flush completes
doing cleanup...

If the above code block runs -- it is called on each append to WAL -- and it finds an empty
map for oldest sequence ids, it will add the current sequenceid... which will likely be far
in excess of what the actual oldest sequence id is.

If the heartbeat runs to report these new sequenceids to the master just before we crash,
then master will have these advanced seqids and they will be used when split WAL runs.

Let me prove above is for sure the problem.




> Splitting WALs, we are filtering out too many edits -> DATALOSS
> ---------------------------------------------------------------
>
>                 Key: HBASE-13811
>                 URL: https://issues.apache.org/jira/browse/HBASE-13811
>             Project: HBase
>          Issue Type: Bug
>          Components: wal
>            Reporter: stack
>            Priority: Critical
>
> I've been running ITBLLs against branch-1 around HBASE-13616 (move of ServerShutdownHandler
to pv2). I have come across an instance of dataloss. My patch for HBASE-13616 was in place
so can only think it the cause (but cannot see how). When we split the logs, we are skipping
legit edits. Digging.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message