hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7964) Add support for async edit logging
Date Wed, 02 Nov 2016 10:32:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628513#comment-15628513
] 

Hadoop QA commented on HDFS-7964:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 20s{color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 11 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 16s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 51s{color}
| {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  3s{color} |
{color:green} branch-2.7 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 20s{color} |
{color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 54s{color}
| {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 45s{color} |
{color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 36s{color}
| {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 52s{color} |
{color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 20s{color} |
{color:green} branch-2.7 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 18s{color} |
{color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  9s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 31s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 11s{color} |
{color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 11s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 10s{color} |
{color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 10s{color} | {color:green}
the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  0m 44s{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 12 new + 1219 unchanged
- 38 fixed = 1231 total (was 1257) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 24s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 25s{color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  0s{color} | {color:red}
The patch has 3712 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>.
Refer https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  1m 37s{color} | {color:red}
The patch 197 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 22s{color} | {color:red}
hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 23s{color} |
{color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  9s{color} |
{color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 25s{color} | {color:red}
hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  1m 19s{color} | {color:red}
bkjournal in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 25s{color} | {color:red}
The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}255m  1s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Inconsistent synchronization of org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync;
locked 50% of time  Unsynchronized access at FSEditLog.java:50% of time  Unsynchronized access
at FSEditLog.java:[line 676] |
| JDK v1.8.0_111 Failed junit tests | hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead
|
|   | hadoop.hdfs.server.namenode.TestNameNodeRecovery |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.namenode.TestAuditLogger |
|   | hadoop.hdfs.server.namenode.TestBackupNode |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
|   | hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead |
| JDK v1.8.0_111 Timed out junit tests | org.apache.hadoop.hdfs.server.namenode.TestEditLog
|
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogRace |
|   | org.apache.hadoop.hdfs.server.namenode.TestAuditLogs |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogAutoroll |
|   | org.apache.hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir |
| JDK v1.7.0_111 Failed junit tests | hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir
|
|   | hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead |
|   | hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints |
|   | hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM |
|   | hadoop.contrib.bkjournal.TestBookKeeperJournalManager |
|   | hadoop.hdfs.server.namenode.TestNameNodeRecovery |
|   | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
|   | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
|   | hadoop.hdfs.server.datanode.TestBlockReplacement |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.contrib.bkjournal.TestBookKeeperAsHASharedDir |
|   | hadoop.contrib.bkjournal.TestBookKeeperSpeculativeRead |
|   | hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints |
|   | hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM |
|   | hadoop.contrib.bkjournal.TestBookKeeperJournalManager |
| JDK v1.7.0_111 Timed out junit tests | org.apache.hadoop.hdfs.server.namenode.TestEditLog
|
|   | org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogJournalFailures |
|   | org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogRace |
|   | org.apache.hadoop.hdfs.server.namenode.TestAuditLogs |
|   | org.apache.hadoop.hdfs.server.namenode.TestEditLogAutoroll |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:c420dfe |
| JIRA Issue | HDFS-7964 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12836510/HDFS-7964-branch-2.7.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux f6d51583e6b4 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.7 / 49f9836 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
|
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/whitespace-eol.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/whitespace-tabs.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_111.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal-jdk1.7.0_111.txt
|
| JDK v1.7.0_111  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/testReport/
|
| asflicense | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/17377/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add support for async edit logging
> ----------------------------------
>
>                 Key: HDFS-7964
>                 URL: https://issues.apache.org/jira/browse/HDFS-7964
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 2.0.2-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>             Fix For: 2.8.0, 2.9.0, 3.0.0-alpha1
>
>         Attachments: HDFS-7964-branch-2.7.patch, HDFS-7964-branch-2.8.0.patch, HDFS-7964-rebase.patch,
HDFS-7964.patch, HDFS-7964.patch, HDFS-7964.patch, HDFS-7964.patch
>
>
> Edit logging is a major source of contention within the NN.  LogEdit is called within
the namespace write log, while logSync is called outside of the lock to allow greater concurrency.
 The handler thread remains busy until logSync returns to provide the client with a durability
guarantee for the response.
> Write heavy RPC load and/or slow IO causes handlers to stall in logSync.  Although the
write lock is not held, readers are limited/starved and the call queue fills.  Combining an
edit log thread with postponed RPC responses from HADOOP-10300 will provide the same durability
guarantee but immediately free up the handlers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message