Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 2 Mar 2017 01:26:45 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13046863.1488257628000.37874.1488418005406@Atlassian.JIRA>
In-Reply-To: <JIRA.13046863.1488257628000@Atlassian.JIRA>
References: <JIRA.13046863.1488257628000@Atlassian.JIRA> <JIRA.13046863.1488257628636@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17707) New More Accurate TableSkew
 Balancer/Generator
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Mar 2017 01:26:50 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891415#comment-15891415 ] 

Hadoop QA commented on HBASE-17707:
-----------------------------------

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 2s {color} | {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 34s {color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 43s {color} | {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12855486/HBASE-17707-04.patch |
| JIRA Issue | HBASE-17707 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle  compile  |
| uname | Linux 31179cbd1682 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh |
| git revision | master / 613bcb3 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5909/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5909/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.


> New More Accurate TableSkew Balancer/Generator
> ----------------------------------------------
>
>                 Key: HBASE-17707
>                 URL: https://issues.apache.org/jira/browse/HBASE-17707
>             Project: HBase
>          Issue Type: New Feature
>          Components: Balancer
>    Affects Versions: 1.2.0
>         Environment: CentOS Derivative with a derivative of the 3.18.43 kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches.
>            Reporter: Kahlil Oppenheimer
>            Priority: Minor
>              Labels: patch
>         Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch
>
>
> This patch includes new version of the TableSkewCostFunction and a new TableSkewCandidateGenerator.
> The new TableSkewCostFunction computes table skew by counting the minimal number of region moves required for a given table to perfectly balance the table across the cluster (i.e. as if the regions from that table had been round-robin-ed across the cluster). This number of moves is computer for each table, then normalized to a score between 0-1 by dividing by the number of moves required in the absolute worst case (i.e. the entire table is stored on one server), and stored in an array. The cost function then takes a weighted average of the average and maximum value across all tables. The weights in this average are configurable to allow for certain users to more strongly penalize situations where one table is skewed versus where every table is a little bit skewed. To better spread this value more evenly across the range 0-1, we take the square root of the weighted average to get the final value.
> The new TableSkewCandidateGenerator generates region moves/swaps to optimize the above TableSkewCostFunction. It first simply tries to move regions until each server has the right number of regions, then it swaps regions around such that each region swap improves table skew across the cluster.
> We tested the cost function and generator in our production clusters with 100s of TBs of data and 100s of tables across dozens of servers and found both to be very performant and accurate.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)