Return-Path: 
 <mapreduce-issues-return-10033-apmail-hadoop-mapreduce-issues-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org
Received: (qmail 27793 invoked from network); 10 Feb 2010 06:44:49 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3)
  by minotaur.apache.org with SMTP; 10 Feb 2010 06:44:49 -0000
Received: (qmail 88764 invoked by uid 500); 10 Feb 2010 06:44:49 -0000
Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org
Received: (qmail 88694 invoked by uid 500); 10 Feb 2010 06:44:49 -0000
Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:mapreduce-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:mapreduce-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:mapreduce-issues@hadoop.apache.org>
List-Id: <mapreduce-issues.hadoop.apache.org>
Reply-To: mapreduce-issues@hadoop.apache.org
Delivered-To: mailing list mapreduce-issues@hadoop.apache.org
Received: (qmail 88679 invoked by uid 99); 10 Feb 2010 06:44:49 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 06:44:49 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Feb 2010 06:44:48 +0000
Received: from brutus.apache.org (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 08F8229A0011
	for <mapreduce-issues@hadoop.apache.org>;
 Tue,  9 Feb 2010 22:44:28 -0800 (PST)
Message-ID: <1388022980.173281265784268034.JavaMail.jira@brutus.apache.org>
Date: Wed, 10 Feb 2010 06:44:28 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Updated: (MAPREDUCE-1305) Massive performance problem with
 DistCp and -delete
In-Reply-To: <726737098.1261002858091.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/MAPREDUCE-1305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Douglas updated MAPREDUCE-1305:
-------------------------------------

    Attachment: M1305-1.patch

Modified Peter's patch to remove FsShell invocations.

That part isn't actually horrible, performance-wise; it reuses the instance, so while there's certainly avoidable overhead in parsing and whatnot, it's not forking a process or anything too notable. It also supports the Trash, which may be useful/appreciated.

Is supporting Trash useful for DistCp users running with \-delete?

> Massive performance problem with DistCp and -delete
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-1305
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1305
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: distcp
>    Affects Versions: 0.20.1
>            Reporter: Peter Romianowski
>            Assignee: Peter Romianowski
>         Attachments: M1305-1.patch, MAPREDUCE-1305.patch
>
>
> *First problem*
> In org.apache.hadoop.tools.DistCp#deleteNonexisting we serialize FileStatus objects when the path is all we need.
> The performance problem comes from org.apache.hadoop.fs.RawLocalFileSystem.RawLocalFileStatus#write which tries to retrieve file permissions by issuing a "ls -ld <path>" which is painfully slow.
> Changed that to just serialize Path and not FileStatus.
> *Second problem*
> To delete the files we invoke the "hadoop" command line tool with option "-rmr <path>". Again, for each file.
> Changed that to dstfs.delete(path, true)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.