hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8791) block ID-based DN storage layout can be very slow for datanode on ext4
Date Wed, 02 Dec 2015 00:44:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034993#comment-15034993
] 

Hadoop QA commented on HDFS-8791:
---------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 4s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 51s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 39s {color} |
{color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s {color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m 34s {color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 56s {color}
| {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 24m 35s {color} | {color:red}
root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 19s {color} |
{color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m 57s {color} |
{color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 59s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 41s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 41s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 40s {color} |
{color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 40s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m 12s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 45s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red}
The patch has 5 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 24m 50s {color} | {color:red}
root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 29s {color} |
{color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m 0s {color} |
{color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 53s {color} | {color:red}
root in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 56s {color} | {color:red}
root in the patch failed with JDK v1.7.0_85. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 201m 24s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
|   | hadoop.ipc.TestIPC |
|   | hadoop.test.TestTimedOutTestsListener |
| JDK v1.7.0_85 Failed junit tests | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
|   | hadoop.fs.TestFsShellReturnCode |
|   | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12775044/HDFS-8791-trunk-v2-bin.patch
|
| JIRA Issue | HDFS-8791 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux 7fe1bdd9a726 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 830eb25 |
| findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/branch-findbugs-root.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/whitespace-tabs.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/patch-findbugs-root.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
|
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/patch-unit-root-jdk1.7.0_85.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
https://builds.apache.org/job/PreCommit-HDFS-Build/13722/artifact/patchprocess/patch-unit-root-jdk1.7.0_85.txt
|
| JDK v1.7.0_85  Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/testReport/
|
| modules | C: . U: . |
| Max memory used | 116MB |
| Powered by | Apache Yetus   http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/13722/console |


This message was automatically generated.



> block ID-based DN storage layout can be very slow for datanode on ext4
> ----------------------------------------------------------------------
>
>                 Key: HDFS-8791
>                 URL: https://issues.apache.org/jira/browse/HDFS-8791
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.6.0, 2.8.0, 2.7.1
>            Reporter: Nathan Roberts
>            Assignee: Chris Trezzo
>            Priority: Critical
>         Attachments: 32x32DatanodeLayoutTesting-v1.pdf, 32x32DatanodeLayoutTesting-v2.pdf,
HDFS-8791-trunk-v1.patch, HDFS-8791-trunk-v2-bin.patch, HDFS-8791-trunk-v2.patch, HDFS-8791-trunk-v2.patch,
hadoop-56-layout-datanode-dir.tgz
>
>
> We are seeing cases where the new directory layout causes the datanode to basically cause
the disks to seek for 10s of minutes. This can be when the datanode is running du, and it
can also be when it is performing a checkDirs(). Both of these operations currently scan all
directories in the block pool and that's very expensive in the new layout.
> The new layout creates 256 subdirs, each with 256 subdirs. Essentially 64K leaf directories
where block files are placed.
> So, what we have on disk is:
> - 256 inodes for the first level directories
> - 256 directory blocks for the first level directories
> - 256*256 inodes for the second level directories
> - 256*256 directory blocks for the second level directories
> - Then the inodes and blocks to store the the HDFS blocks themselves.
> The main problem is the 256*256 directory blocks. 
> inodes and dentries will be cached by linux and one can configure how likely the system
is to prune those entries (vfs_cache_pressure). However, ext4 relies on the buffer cache to
cache the directory blocks and I'm not aware of any way to tell linux to favor buffer cache
pages (even if it did I'm not sure I would want it to in general).
> Also, ext4 tries hard to spread directories evenly across the entire volume, this basically
means the 64K directory blocks are probably randomly spread across the entire disk. A du type
scan will look at directories one at a time, so the ioscheduler can't optimize the corresponding
seeks, meaning the seeks will be random and far. 
> In a system I was using to diagnose this, I had 60K blocks. A DU when things are hot
is less than 1 second. When things are cold, about 20 minutes.
> How do things get cold?
> - A large set of tasks run on the node. This pushes almost all of the buffer cache out,
causing the next DU to hit this situation. We are seeing cases where a large job can cause
a seek storm across the entire cluster.
> Why didn't the previous layout see this?
> - It might have but it wasn't nearly as pronounced. The previous layout would be a few
hundred directory blocks. Even when completely cold, these would only take a few a hundred
seeks which would mean single digit seconds.  
> - With only a few hundred directories, the odds of the directory blocks getting modified
is quite high, this keeps those blocks hot and much less likely to be evicted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message