hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15400) Use DateTieredCompactor for Date Tiered Compaction
Date Sat, 19 Mar 2016 01:47:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202489#comment-15202489
] 

Hadoop QA commented on HBASE-15400:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 27s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s {color} | {color:green}
master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s {color} |
{color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 56s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s {color} |
{color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s {color} |
{color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 42s {color} | {color:red}
hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s {color} | {color:red}
hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s {color} | {color:red}
hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s {color} | {color:red}
hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 38s {color} | {color:red}
hbase-server: patch generated 1 new + 96 unchanged - 28 fixed = 97 total (was 124) {color}
|
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red}
The patch has 6 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 55s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 50s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 44s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 42s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 38s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 38s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 39s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 35s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 30s {color} | {color:red}
Patch causes 27 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 27s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 15s {color} | {color:red}
hbase-server-jdk1.8.0 with JDK v1.8.0 generated 3 new + 1 unchanged - 0 fixed = 4 total (was
1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s {color} |
{color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 56s {color} | {color:red}
hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 3 new + 1 unchanged - 0 fixed = 4 total
(was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s {color} |
{color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 10s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 22s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12794245/HBASE-15400-15389-v12.patch
|
| JIRA Issue | HBASE-15400 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3a6d683 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 /usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79
|
| findbugs | v3.0.0 |
| mvninstall | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-mvninstall-hbase-server.txt
|
| compile | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-compile-hbase-server-jdk1.8.0.txt
|
| javac | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-compile-hbase-server-jdk1.8.0.txt
|
| compile | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_79.txt
|
| javac | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-compile-hbase-server-jdk1.7.0_79.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/whitespace-tabs.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-findbugs-hbase-server.txt
|
| javadoc | hbase-server-jdk1.8.0: https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/diff-javadoc-javadoc-hbase-server-jdk1.8.0.txt
|
| javadoc | hbase-server-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/diff-javadoc-javadoc-hbase-server-jdk1.7.0_79.txt
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/artifact/patchprocess/patch-unit-hbase-server.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/1077/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Use DateTieredCompactor for Date Tiered Compaction
> --------------------------------------------------
>
>                 Key: HBASE-15400
>                 URL: https://issues.apache.org/jira/browse/HBASE-15400
>             Project: HBase
>          Issue Type: Sub-task
>          Components: Compaction
>            Reporter: Clara Xiong
>            Assignee: Clara Xiong
>             Fix For: 2.0.0
>
>         Attachments: HBASE-15400-15389-v12.patch, HBASE-15400-v1.pa, HBASE-15400.patch
>
>
> When we compact, we can output multiple files along the current window boundaries. There
are two use cases:
> 1. Major compaction: We want to output date tiered store files with data older than max
age archived in trunks of the window size on the higher tier.1. Once a file is old enough
to be out of the range that we do tiered minor compaction, we don't compact them any further.
So they retain the same timespan as they were compacted last time, which is the window size
of the highest tier. Major compaction will touch these files and we want to maintain the same
layout.
> 2. Bulk load files and the old file generated by major compaction before upgrading to
DTCP.
> Pros: 
> 1. Restore locality, process versioning, updates and deletes while maintaining the tiered
layout.
> 2. The best way to fix a skewed layout.
>  
> This work is based on a prototype of DateTieredCompactor from HBASE-15389 and focused
on the part to meet needs for these two use cases while supporting others. I have to call
out a few design decisions:
> 1. We only want to output the files along all windows for major compaction. And we want
to output multiple files older than max age in the trunks of the maximum tier window size
determined by base window size, windows per tier and max age.
> 2. For minor compaction, we don't want to output too many files, which will remain around
because of current restriction of contiguous compaction by seq id. I will only output two
files if all the files in the windows are being combined, one for the data within window and
the other for the out-of-window tail. If there is any file in the window excluded from compaction,
only one file will be output from compaction. When the windows are promoted, the situation
of out of order data will gradually improve. For the incoming window, we need to accommodate
the case with user-specified future data.
> 3. We have to pass the boundaries with the list of store file as a complete time snapshot
instead of two separate calls because window layout is determined by the time the computation
is called. So we will need new type of compaction request. 
> 4. Since we will assign the same seq id for all output files, we need to sort by maxTimestamp
subsequently. Right now all compaction policy gets the files sorted for StoreFileManager which
sorts by seq id and other criteria. I will use this order for DTCP only, to avoid impacting
other compaction policies. 
> 5. We need some cleanup of current design of StoreEngine and CompactionPolicy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message