hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12590) A solution for data skew in HBase-Mapreduce Job
Date Wed, 04 Oct 2017 04:06:05 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16190768#comment-16190768
] 

Hudson commented on HBASE-12590:
--------------------------------

SUCCESS: Integrated in Jenkins build HBase-1.4 #940 (See [https://builds.apache.org/job/HBase-1.4/940/])
HBASE-16894 Create more than 1 split per region, generalize HBASE-12590 (apurtell: rev cbbcb2db2f0a94382cb33fef826cbf1a00b5de6e)
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/namespace/TestNamespaceAuditor.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
* (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan1.java


> A solution for data skew in HBase-Mapreduce Job
> -----------------------------------------------
>
>                 Key: HBASE-12590
>                 URL: https://issues.apache.org/jira/browse/HBASE-12590
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>            Reporter: Weichen Ye
>            Assignee: Weichen Ye
>             Fix For: 2.0.0
>
>         Attachments: A Solution for Data Skew in HBase-MapReduce Job (Version2).pdf,
A Solution for Data Skew in HBase-MapReduce Job (Version3).pdf, HBase-12590-v1.patch, HBase-12590-v2.patch,
HBASE-12590-v3.patch, HBASE-12590-v4.patch
>
>
> 1, Motivation
> In production environment, data skew is a very common case. A HBase table may contains
a lot of small regions and several large regions. Small regions waste a lot of computing resources.
If we use a job to scan a table with 3000 small regions, we need a job with 3000 mappers.
Large regions always block the job. If in a 100-region table, one region is far large then
the other 99 regions. When we run a job with the table as input, 99 mappers will be completed
very quickly, and then we need to wait for the last mapper for a long time.
> 2, Configuration
> Add three new configuration 
> hbase.mapreduce.input.autobalance = true means enabling the “auto balance” in HBase-MapReduce
jobs. The default value is false. 
> hbase.mapreduce.input.autobalance.maxskewratio= 3 (default is 3). If a region size is
larger than 3x average region size, treat the region as “proportionately too large”.
> hbase.table.row.textkey  = true means the row key is text. False means binary row key.
It is used to find the mid row key in large region. The default value is true. 
> If (region size >= average size*ratio) :  cut the region into two MR input splits
> If (average size <= region size < average size*ratio) : one region as one MR input
split
> If (sum of several continuous regions size < average size): combine these regions
into one MR input split.
> Example:
> In attachment
> Welcome to the Review Board.
> https://reviews.apache.org/r/28494/diff/#



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message