hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
Date Wed, 09 Mar 2011 06:04:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004383#comment-13004383
] 

Hairong Kuang commented on HADOOP-7133:
---------------------------------------

Matt, this optimization is fantastic. I have ported it to our internal branch. Together with
parallel upgrades, it cut the DN upgrade time from 1 hour 40 minutes to 1 minute! Awesome
job!

+1. The patch looks good. One minor optimization is that you could use ThreadLocal for shell
commands. This will remove the need to create a buffer for each shell command to run.

> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(),
so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-7133
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7133
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>    Affects Versions: 0.20.2
>            Reporter: Matt Foley
>            Assignee: Matt Foley
>             Fix For: 0.22.0
>
>         Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so
we call it once per directory instead of once per file" requires coordinated change in COMMON
and HDFS.  This is the COMMON portion, submitted here under a separate bug to activate the
automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS.  It requires coordinated commit
of the HDFS portion of the patch in HDFS-1445.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message