hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9759) IntegrationTestBulkLoad random number collision
Date Tue, 15 Oct 2013 04:37:43 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794875#comment-13794875
] 

Hadoop QA commented on HBASE-9759:
----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12608424/hbase-9759_v1.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 3 new or modified
tests.

    {color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 1.0 profile.

    {color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 2.0 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

    {color:red}-1 site{color}.  The patch appears to cause mvn site goal to fail.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7542//console

This message is automatically generated.

> IntegrationTestBulkLoad random number collision
> -----------------------------------------------
>
>                 Key: HBASE-9759
>                 URL: https://issues.apache.org/jira/browse/HBASE-9759
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 0.98.0, 0.96.1
>
>         Attachments: hbase-9759_v1.patch
>
>
> ITBL failed recently in our test harness. Inspecting the failure made me believe that
the only reason that particular failure might have happened is that there is a collision in
random longs generated by the test. 
> The test creates 50 mappers by default, and each mapper writes a 500K random rows starting
with row = 0. By default there are 5 iterations.
> The check job outputs these counters: 
> {code}
> 2013-10-13 07:48:01,134 Map input records=124999751
> 2013-10-13 07:48:01,134 Map output records=124999999
> {code}
> The number of input records seems fine because
> {code}
> 124999751 = 1 + 5 * (0.5M - 1) * 50
> {code}
> 5 = num iterations, 0.5M = num rows, 50 = num mappers, and 1 is for row =0 which every
chain writes to. 
> Output records should be 125M, however we see one cell missing. Since the map input records
matches expected number of distinct rows, I suspect that row = 0 had a collision. 
> In one of the generate jobs, we can see that the reducer output count does not match
the reducer input count. Given that we are using KVSortReducer, this confirms that there is
a collision in KeyValues received by this task.
> {code}
> 2013-10-13 06:48:12,738 Reduce input records=75000000
> 2013-10-13 06:48:12,738 Reduce output records=74999997
> {code}
> The count is off by 3 because we are writing 3 columns per row. 
> My only theory for explaining this is that we had a collision in chainId's or one of
the chains reused row = 0 as the next row. 
> This is similar to HBASE-8700, however, in this the probability is much much much lower.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message