hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1445) Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
Date Fri, 11 Mar 2011 19:15:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005792#comment-13005792
] 

Hadoop QA commented on HDFS-1445:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12470696/HDFS-1445-trunk.v22_hdfs_2-of-2.patch
  against trunk revision 1080380.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit
warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestFileConcurrentReader

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/250//console

This message is automatically generated.

> Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory
instead of once per file
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1445
>                 URL: https://issues.apache.org/jira/browse/HDFS-1445
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: data-node
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1445-trunk.v22_hdfs_2-of-2.patch
>
>
> It was a bit of a puzzle why we can do a full scan of a disk in about 30 seconds during
FSDir() or getVolumeMap(), but the same disk took 11 minutes to do Upgrade replication via
hardlinks.  It turns out that the org.apache.hadoop.fs.FileUtil.createHardLink() method does
an outcall to Runtime.getRuntime().exec(), to utilize native filesystem hardlink capability.
 So it is forking a full-weight external process, and we call it on each individual file to
be replicated.
> As a simple check on the possible cost of this approach, I built a Perl test script (under
Linux on a production-class datanode).  Perl also uses a compiled and optimized p-code engine,
and it has both native support for hardlinks and the ability to do "exec".  
> -  A simple script to create 256,000 files in a directory tree organized like the Datanode,
took 10 seconds to run.
> -  Replicating that directory tree using hardlinks, the same way as the Datanode, took
12 seconds using native hardlink support.
> -  The same replication using outcalls to exec, one per file, took 256 seconds!
> -  Batching the calls, and doing 'exec' once per directory instead of once per file,
took 16 seconds.
> Obviously, your mileage will vary based on the number of blocks per volume.  A volume
with less than about 4000 blocks will have only 65 directories.  A volume with more than 4K
and less than about 250K blocks will have 4200 directories (more or less).  And there are
two files per block (the data file and the .meta file).  So the average number of files per
directory may vary from 2:1 to 500:1.  A node with 50K blocks and four volumes will have 25K
files per volume, or an average of about 6:1.  So this change may be expected to take it down
from, say, 12 minutes per volume to 2.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message