hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19528) Major Compaction Tool
Date Sat, 03 Feb 2018 09:32:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351343#comment-16351343
] 

Hadoop QA commented on HBASE-19528:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 46s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  0s{color}
| {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 2 new or modified test files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m  0s{color}
| {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 40s{color} |
{color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 38s{color} |
{color:green} branch-1 passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 17s{color}
| {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  3m 52s{color}
| {color:green} branch has no errors when building our shaded downstream artifacts. {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 13s{color} |
{color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 32s{color} |
{color:green} branch-1 passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 36s{color} |
{color:green} branch-1 passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 34s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 39s{color} |
{color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 39s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 38s{color} |
{color:green} the patch passed with JDK v1.7.0_171 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 38s{color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 17s{color} | {color:red}
hbase-server: The patch generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green}  2m 32s{color}
| {color:green} patch has no errors when building our shaded downstream artifacts. {color}
|
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red}  3m 29s{color} | {color:red}
The patch causes 44 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red}  4m 23s{color} | {color:red}
The patch causes 44 errors with Hadoop v2.5.2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 26s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 33s{color} |
{color:green} the patch passed with JDK v1.8.0_162 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 36s{color} |
{color:green} the patch passed with JDK v1.7.0_171 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m 17s{color} | {color:green}
hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 19s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 43s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:36a7029 |
| JIRA Issue | HBASE-19528 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909086/HBASE-19528.v2.branch-1.patch
|
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  shadedjars  hadoopcheck  hbaseanti
 checkstyle  compile  |
| uname | Linux 6b519f89656f 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19 15:49:21 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-1 / a55f2c7 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_171 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_162 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_171
|
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/11373/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11373/testReport/ |
| Max. process+thread count | 3782 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11373/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Major Compaction Tool 
> ----------------------
>
>                 Key: HBASE-19528
>                 URL: https://issues.apache.org/jira/browse/HBASE-19528
>             Project: HBase
>          Issue Type: New Feature
>            Reporter: churro morales
>            Assignee: churro morales
>            Priority: Major
>             Fix For: 2.0.0-beta-2
>
>         Attachments: 0001-HBASE-19528-Major-Compaction-Tool-ADDENDUM.patch, HBASE-19528.branch-1.patch,
HBASE-19528.patch, HBASE-19528.v1.branch-1.patch, HBASE-19528.v1.patch, HBASE-19528.v2.branch-1.patch,
HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch, HBASE-19528.v2.branch-1.patch,
HBASE-19528.v8.patch
>
>
> The basic overview of how this tool works is:
> Parameters:
>     Table
>     Stores
>     ClusterConcurrency
>     Timestamp
> So you input a table, desired concurrency and the list of stores you wish to major compact.
 The tool first checks the filesystem to see which stores need compaction based on the timestamp
you provide (default is current time).  It takes that list of stores that require compaction
and executes those requests concurrently with at most N distinct RegionServers compacting
at a given time.  Each thread waits for the compaction to complete before moving to the next
queue.  If a region split, merge or move happens this tool ensures those regions get major
compacted as well. 
> This helps us in two ways, we can limit how much I/O bandwidth we are using for major
compaction cluster wide and we are guaranteed after the tool completes that all requested
compactions complete regardless of moves, merges and splits. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message