hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9381) When same block came for replication for Striped mode, we can move that block to PendingReplications
Date Thu, 26 Nov 2015 00:42:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027905#comment-15027905
] 

Hadoop QA commented on HDFS-9381:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 49s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s {color} |
{color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s {color} | {color:green}
trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s {color} | {color:green}
trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 17s {color} | {color:red}
hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66 with JDK v1.8.0_66 generated 2 new issues (was
33, now 33). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s {color} |
{color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 6s {color} | {color:red}
hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85 with JDK v1.7.0_85 generated 2 new issues (was
35, now 35). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s {color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s {color} | {color:red}
Patch generated 1 new checkstyle issues in hadoop-hdfs-project/hadoop-hdfs (total was 169,
now 169). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 21s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s {color} | {color:green}
the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 44s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 1s {color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s {color} | {color:red}
Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 177m 30s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
|
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
|   | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
|   | hadoop.hdfs.server.datanode.TestBlockScanner |
| JDK v1.7.0_85 Failed junit tests | hadoop.hdfs.TestReadStripedFileWithDecoding |
|   | hadoop.hdfs.qjournal.client.TestQuorumJournalManager |
|   | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12774414/HDFS-9381-03.patch
|
| JIRA Issue | HDFS-9381 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 62877edd9dc2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e3d6739 |
| findbugs | v3.0.0 |
| javac | hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
|
| javac | hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_85.txt
|
| JDK v1.7.0_85  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/13666/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-HDFS-Build/13666/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Max memory used | 76MB |
| Powered by | Apache Yetus   http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/13666/console |


This message was automatically generated.



> When same block came for replication for Striped mode, we can move that block to PendingReplications
> ----------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-9381
>                 URL: https://issues.apache.org/jira/browse/HDFS-9381
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: erasure-coding, namenode
>    Affects Versions: 3.0.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HDFS-9381-02.patch, HDFS-9381-03.patch, HDFS-9381.00.patch, HDFS-9381.01.patch
>
>
> Currently I noticed that we are just returning null if block already exists in pendingReplications
in replication flow for striped blocks.
> {code}
> if (block.isStriped()) {
>       if (pendingNum > 0) {
>         // Wait the previous recovery to finish.
>         return null;
>       }
> {code}
>  Here if we just return null and if neededReplications contains only fewer blocks(basically
by default if less than numliveNodes*2), then same blocks can be picked again from neededReplications
from next loop as we are not removing element from neededReplications. Since this replication
process need to take fsnamesystmem lock and do, we may spend some time unnecessarily in every
loop. 
> So my suggestion/improvement is:
>  Instead of just returning null, how about incrementing pendingReplications for this
block and remove from neededReplications? and also another point to consider here is, to add
into pendingReplications, generally we need target and it is nothing but to which node we
issued replication command. Later when after replication success and DN reported it, block
will be removed from pendingReplications from NN addBlock. 
>  So since this is newly picked block from neededReplications, we would not have selected
target yet. So which target to be passed to pendingReplications if we add this block? One
Option I am thinking is, how about just passing srcNode itself as target for this special
condition? So, anyway if the block is really missed, srcNode will not report it. So this block
will not be removed from pending replications, so that when it is timed out, it will be considered
for replication again and that time it will find actual target to replicate while processing
as part of regular replication flow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message