hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10501) Make IncreasingToUpperBoundRegionSplitPolicy configurable
Date Mon, 17 Feb 2014 03:32:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902929#comment-13902929
] 

Hadoop QA commented on HBASE-10501:
-----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12629296/10501-trunk.txt
  against trunk revision .
  ATTACHMENT ID: 12629296

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 6 new or modified
tests.

    {color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 1.0 profile.

    {color:green}+1 hadoop1.1{color}.  The patch compiles against the hadoop 1.1 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:red}-1 findbugs{color}.  The patch appears to introduce 1 new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/8723//console

This message is automatically generated.

> Make IncreasingToUpperBoundRegionSplitPolicy configurable
> ---------------------------------------------------------
>
>                 Key: HBASE-10501
>                 URL: https://issues.apache.org/jira/browse/HBASE-10501
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Lars Hofhansl
>            Assignee: Lars Hofhansl
>            Priority: Critical
>             Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.17
>
>         Attachments: 10501-0.94-v2.txt, 10501-0.94-v3.txt, 10501-0.94-v4.txt, 10501-0.94.txt,
10501-trunk.txt
>
>
> During some (admittedly artificial) load testing we found a large amount split activity,
which we tracked down the IncreasingToUpperBoundRegionSplitPolicy.
> The current logic is this (from the comments):
> "regions that are on this server that all are of the same table, squared, times the region
flush size OR the maximum region split size, whichever is smaller"
> So with a flush size of 128mb and max file size of 20gb, we'd need 13 region of the same
table on an RS to reach the max size.
> With 10gb file sized it is still 9 regions of the same table.
> Considering that the number of regions that an RS can carry is limited and there might
be multiple tables, this should be more configurable.
> I think the squaring is smart and we do not need to change it.
> We could
> * Make the start size configurable and default it to the flush size
> * Add multiplier for the initial size, i.e. start with n * flushSize
> * Also change the default to start with 2*flush size
> Of course one can override the default split policy, but these seem like simple tweaks.
> Or we could instead set the goal of how many regions of the same table would need to
be present in order to reach the max size. In that case we'd start with maxSize/goal^2. So
if max size is 20gb and the goal is three we'd start with 20g/9 = 2.2g for the initial region
size.
> [~stack], I'm especially interested in your opinion.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message