hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
Date Thu, 05 Jul 2012 06:53:35 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406881#comment-13406881
] 

Hadoop QA commented on HBASE-6311:
----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12535148/HBASE-6311v2.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 4 new or modified tests.

    +1 hadoop2.0.  The patch compiles against the hadoop 2.0 profile.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The applied patch generated 5 javac compiler warnings (more than the trunk's
current 4 warnings).

    -1 findbugs.  The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

     -1 core tests.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.regionserver.TestAtomicOperation
                  org.apache.hadoop.hbase.replication.TestReplication

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2325//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2325//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2325//console

This message is automatically generated.
                
> Data error after majorCompaction caused by keeping MVCC for opened scanners
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-6311
>                 URL: https://issues.apache.org/jira/browse/HBASE-6311
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.94.0
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Blocker
>             Fix For: 0.96.0, 0.94.1
>
>         Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, HBASE-6311v2.patch
>
>
> It is a big problem we found in 0.94, and you could reproduce the problem in Trunk using
the test case I uploaded.
> When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC for opened
scanners;
> However,It will make data mistake after majorCompaction because we will skip delete type
KV but keep the put type kv in the compacted storefile.
> The following is the reason from code:
> In StoreFileScanner, enforceMVCC is false when compaction, so we could read the delete
type KV,
> However, we will skip this delete type KV in ScanQueryMatcher because following code
> {code}
> if (kv.isDelete())
> {
> ...
>  if (includeDeleteMarker
>             && kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
>           System.out.println("add deletes,maxReadPointToTrackVersions="
>               + maxReadPointToTrackVersions);
>           this.deletes.add(bytes, offset, qualLength, timestamp, type);
>         }
> ...
> }
> {code}
> Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
> and kv.getMemstoreTS() > maxReadPointToTrackVersions 
> So we won't add this to DeleteTracker.
> Why test case passed if remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
> Because in the StoreFileScanner#skipKVsNewerThanReadpoint
> {code}
> if (cur.getMemstoreTS() <= readPoint) {
>       cur.setMemstoreTS(0);
>     }
> {code}
> So if we remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
> Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add it to
DeleteTracker in ScanQueryMatcher 
> Solution:
> We use smallestReadPoint of region when compaction to keep MVCC for OPENED scanner, So
we should retain delete type kv in output in the case(Already deleted KV is retained in output
to make old opened scanner could read this KV) even if it is a majorcompaction.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message