hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chunhui shen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-6311) Data error after compaction because of scanners
Date Tue, 03 Jul 2012 05:25:03 GMT

     [ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

chunhui shen updated HBASE-6311:
--------------------------------

    Description: 
It is a big problem we found in 0.94, and you could reproduce the problem in Trunk using the
test case I uploaded.

When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC for opened scanners;
However,It will make data mistake after compaction because we will skip delete type KV but
keep the put type kv.

The following is the reason from code:
In StoreFileScanner, enforceMVCC is false when compaction, so we could read the delete type
KV,
However, we will skip this delete type KV in ScanQueryMatcher because following code

{code}
if (kv.isDelete())
{
...
 if (includeDeleteMarker
            && kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
          System.out.println("add deletes,maxReadPointToTrackVersions="
              + maxReadPointToTrackVersions);
          this.deletes.add(bytes, offset, qualLength, timestamp, type);
        }
...
}
{code}

Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
and kv.getMemstoreTS() > maxReadPointToTrackVersions 
So we won't add this to DeleteTracker.

Why test case passed if remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);

Because in the StoreFileScanner#skipKVsNewerThanReadpoint
{code}
if (cur.getMemstoreTS() <= readPoint) {
      cur.setMemstoreTS(0);
    }
{code}
So if we remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add it to DeleteTracker
in ScanQueryMatcher 




  was:
It is a big problem found in 0.94, and you could reproduce the problem in Trunk using the
test case I uploaded:

When we do compaction, we will use region.getSmallestReadPoint();
It will make data mistake after compaction because we will skip delete type KV but keep the
put type kv.

The following is the reason:
In StoreFileScanner, enforceMVCC is false when compaction, so we could read the delete type
KV,
However, we will skip this delete type KV in ScanQueryMatcher because following code

{code}
if (kv.isDelete())
{
...
 if (includeDeleteMarker
            && kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
          System.out.println("add deletes,maxReadPointToTrackVersions="
              + maxReadPointToTrackVersions);
          this.deletes.add(bytes, offset, qualLength, timestamp, type);
        }
...
}
{code}

Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
and kv.getMemstoreTS() > maxReadPointToTrackVersions 
So we won't add this to DeleteTracker.

Why test case passed if remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);

Because in the StoreFileScanner#skipKVsNewerThanReadpoint
{code}
if (cur.getMemstoreTS() <= readPoint) {
      cur.setMemstoreTS(0);
    }
{code}
So if we remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add it to DeleteTracker
in ScanQueryMatcher 




    
> Data error after compaction because of scanners
> -----------------------------------------------
>
>                 Key: HBASE-6311
>                 URL: https://issues.apache.org/jira/browse/HBASE-6311
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.94.0
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Blocker
>         Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch
>
>
> It is a big problem we found in 0.94, and you could reproduce the problem in Trunk using
the test case I uploaded.
> When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC for opened
scanners;
> However,It will make data mistake after compaction because we will skip delete type KV
but keep the put type kv.
> The following is the reason from code:
> In StoreFileScanner, enforceMVCC is false when compaction, so we could read the delete
type KV,
> However, we will skip this delete type KV in ScanQueryMatcher because following code
> {code}
> if (kv.isDelete())
> {
> ...
>  if (includeDeleteMarker
>             && kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
>           System.out.println("add deletes,maxReadPointToTrackVersions="
>               + maxReadPointToTrackVersions);
>           this.deletes.add(bytes, offset, qualLength, timestamp, type);
>         }
> ...
> }
> {code}
> Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
> and kv.getMemstoreTS() > maxReadPointToTrackVersions 
> So we won't add this to DeleteTracker.
> Why test case passed if remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
> Because in the StoreFileScanner#skipKVsNewerThanReadpoint
> {code}
> if (cur.getMemstoreTS() <= readPoint) {
>       cur.setMemstoreTS(0);
>     }
> {code}
> So if we remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
> Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add it to
DeleteTracker in ScanQueryMatcher 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message