Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 4 Oct 2017 04:06:05 +0000 (UTC)
From: "Hudson (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13014022.1476990413000.266979.1507089965045@Atlassian.JIRA>
In-Reply-To: <JIRA.13014022.1476990413000@Atlassian.JIRA>
References: <JIRA.13014022.1476990413000@Atlassian.JIRA> <JIRA.13014022.1476990413449@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-16894) Create more than 1 split per
 region, generalize HBASE-12590
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 04 Oct 2017 04:06:16 -0000


    [ https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190767#comment-16190767 ] 

Hudson commented on HBASE-16894:
--------------------------------

SUCCESS: Integrated in Jenkins build HBase-1.4 #940 (See [https://builds.apache.org/job/HBase-1.4/940/])
HBASE-16894 Create more than 1 split per region, generalize HBASE-12590 (apurtell: rev cbbcb2db2f0a94382cb33fef826cbf1a00b5de6e)
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/namespace/TestNamespaceAuditor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java


> Create more than 1 split per region, generalize HBASE-12590
> -----------------------------------------------------------
>
>                 Key: HBASE-16894
>                 URL: https://issues.apache.org/jira/browse/HBASE-16894
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0, 2.0.0-alpha-2
>            Reporter: Enis Soztutar
>            Assignee: Yi Liang
>             Fix For: 2.0.0, 3.0.0, 1.4.0, 1.5.0
>
>         Attachments: HBASE-16894.branch-1.patch, HBASE-16894.master.patch, HBASE-16894-V2-master.patch, HBASE-16894-V3-master.patch, ImplementaionAndSomeQuestion.docx
>
>
> A common request from users is to be able to better control how many map tasks are created per region. Right now, it is always 1 region = 1 input split = 1 map task. Same goes for Spark since it uses the TIF. With region sizes as large as 50 GBs, it is desirable to be able to create more than 1 split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input splits
> If (average size <= region size < average size*ratio) : one region as one MR input split
> If (sum of several continuous regions size < average size * ratio): combine these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse HBASE-12590 into creating more than 1 split task per region, it is not ideal. But there is no way to create more with the patch as it is. For example we cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in HBASE-12590, and make it so that the client can specify the desired num of mappers, or desired split size, and the TIF generates the splits based on the current region sizes very similar to the algorithm in HBASE-12590, but a more generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats table which is used for exactly this purpose. Right now, the region can be split into powers of two assuming uniform distribution within the region. 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)