hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Liangliang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14004) Inconsistency between Memstore and WAL
Date Wed, 01 Jul 2015 13:43:05 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

He Liangliang updated HBASE-14004:
----------------------------------
    Description: 
Looks like the current write path can cause inconsistency between memstore/hfile and WAL which
cause the slave cluster has more data than the master cluster.

The simplified write path looks like:
1. insert record into Memstore
2. write record to WAL
3. sync WAL
4. rollback Memstore if 3 fails

It's possible that the HDFS sync RPC call fails, but the data is already  (may partially)
transported to the DNs which finally get persisted. As a result, the handler will rollback
the Memstore and the later flushed HFile will also skip this record.




  was:
We encountered data inconsistency issue between master and slave clusters. Looks like the
current write path can cause inconsistency between memstore/hfile and WAL which cause the
slave cluster has more data than the master cluster.

The simplified write path looks like:
1. insert record into Memstore
2. write record to WAL
3. sync WAL
4. rollback Memstore if 3 fails

In step 3, the sync operation can fail in two case:
1. The caller disconnected the socket due to timeout or any other client crashes
2. The underlying HDFS sync operation can fails

For case 1, the caller may disconnect the socket (for example, timeout due to slow sync),
but the sync can finally succeed because it's an asynchronous operation. As a result, the
handler will rollback the Memstore and the later flushed HFile will also skip this record.
For case 2, it's possible that the HDFS sync RPC call fails, but the data is already  (may
partially) transported to the DNs which finally get persisted.

0.94 code fails in both cases, and looks like the trunk code suffered only case 2.



> Inconsistency between Memstore and WAL
> --------------------------------------
>
>                 Key: HBASE-14004
>                 URL: https://issues.apache.org/jira/browse/HBASE-14004
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>            Reporter: He Liangliang
>            Priority: Critical
>
> Looks like the current write path can cause inconsistency between memstore/hfile and
WAL which cause the slave cluster has more data than the master cluster.
> The simplified write path looks like:
> 1. insert record into Memstore
> 2. write record to WAL
> 3. sync WAL
> 4. rollback Memstore if 3 fails
> It's possible that the HDFS sync RPC call fails, but the data is already  (may partially)
transported to the DNs which finally get persisted. As a result, the handler will rollback
the Memstore and the later flushed HFile will also skip this record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message