hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8131) Implement a space balanced block placement policy
Date Thu, 14 May 2015 12:43:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543620#comment-14543620
] 

Hadoop QA commented on HDFS-8131:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | patch |   0m  1s | The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
|
| {color:blue}0{color} | pre-patch |  14m 34s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 29s | There were no new javac warning messages. |
| {color:red}-1{color} | javadoc |   9m 39s | The applied patch generated  1  additional warning
messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   2m 15s | The applied patch generated  16 new checkstyle
issues (total was 459, now 475). |
| {color:red}-1{color} | whitespace |   0m  0s | The patch has 1  line(s) that end in whitespace.
Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   3m  4s | The patch does not introduce any new Findbugs
(version 2.0.3) warnings. |
| {color:green}+1{color} | native |   3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 168m  5s | Tests failed in hadoop-hdfs. |
| | | 210m 51s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.tools.TestHdfsConfigFields |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12732816/HDFS-8131-v3.diff |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0daa5ad |
| javadoc | https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |  https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/10972/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/10972/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/10972/console |


This message was automatically generated.

> Implement a space balanced block placement policy
> -------------------------------------------------
>
>                 Key: HDFS-8131
>                 URL: https://issues.apache.org/jira/browse/HDFS-8131
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Liu Shaohui
>            Assignee: Liu Shaohui
>            Priority: Minor
>              Labels: BlockPlacementPolicy
>             Fix For: 3.0.0
>
>         Attachments: HDFS-8131-v1.diff, HDFS-8131-v2.diff, HDFS-8131-v3.diff, balanced.png
>
>
> The default block placement policy will choose datanodes for new blocks randomly, which
will result in unbalanced space used percent among datanodes after an cluster expansion. The
old datanodes always are in high used percent of space and new added ones are in low percent.
> Through we can used the external balance tool to balance the space used rate, it will
cost extra network IO and it's not easy to control the balance speed.
> An easy solution is to implement an balanced block placement policy which will choose
low used percent datanodes for new blocks with a little high possibility. In a not long term,
the used percent of datanodes will trend to be balanced.
> Suggestions and discussions are welcomed. Thanks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message