hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11882) Row level consistency may not be maintained with bulk load and compaction
Date Wed, 03 Sep 2014 18:29:52 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14120216#comment-14120216

Hadoop QA commented on HBASE-11882:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision .
  ATTACHMENT ID: 12666066

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 3 new or modified

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

     {color:red}-1 core tests{color}.  The patch failed these unit tests:

     {color:red}-1 core zombie tests{color}.  There are 2 zombie test(s): 	at org.apache.hadoop.hbase.client.TestHCM.testClusterStatus(TestHCM.java:250)
	at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:3813)

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/10695//console

This message is automatically generated.

> Row level consistency may not be maintained with bulk load and compaction
> -------------------------------------------------------------------------
>                 Key: HBASE-11882
>                 URL: https://issues.apache.org/jira/browse/HBASE-11882
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.99.0, 2.0.0
>            Reporter: Jerry He
>            Assignee: Jerry He
>            Priority: Critical
>             Fix For: 0.99.0, 2.0.0
>         Attachments: HBASE-11882-master-v1.patch, HBASE-11882-master-v2.patch, TestHRegionServerBulkLoad.java.patch
> While looking into the TestHRegionServerBulkLoad failure for HBASE-11772, I found the
root cause is that row level atomicity may not be maintained with bulk load together with
> TestHRegionServerBulkLoad is used to test bulk load atomicity. The test uses multiple
threads to do bulk load and scan continuously and do compactions periodically. 
> It verifies row level data is always consistent across column families.
> After HBASE-11591, we added readpoint checks for bulkloaded data using the seqId at the
time of bulk load. Now a scanner will not see the data from a bulk load if the scanner's readpoint
is earlier than the bulk load seqId.
> Previously, the atomic bulk load result is visible immediately to all scanners.
> The problem is with compaction after bulk load. Compaction does not lock the region and
it is done one store (column family) at a time. It also compact away the seqId marker of bulk
> Here is an event sequence where the row level consistency is broken.
> 1. A scanner is started to scan a region with cf1 and cf2. The readpoint is 10.
> 2. There is a bulk load that loads into cf1 and cf2. The bulk load seqId is 11. Bulk
load is guarded by region write lock. So it is atomic.
> 3. There is a compaction that compacts cf1. It compacts away the seqId marker of the
bulk load.
> 4. The scanner tries to next to row-1001. It gets the bulk load data for cf1 since there
is no seqId preventing it.  It does not get the bulk load data for cf2 since the scanner's
readpoint (10) is less than the bulk load seqId (11).
> Now the row level consistency is broken in this case.

This message was sent by Atlassian JIRA

View raw message