Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B5A18186AF for ; Sun, 10 Jan 2016 23:05:53 +0000 (UTC) Received: (qmail 88724 invoked by uid 500); 10 Jan 2016 23:05:49 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 88619 invoked by uid 500); 10 Jan 2016 23:05:49 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 88608 invoked by uid 99); 10 Jan 2016 23:05:48 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jan 2016 23:05:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4711C180562 for ; Sun, 10 Jan 2016 23:05:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id KhJ4ncz62f72 for ; Sun, 10 Jan 2016 23:05:40 +0000 (UTC) Received: from mail-ig0-f175.google.com (mail-ig0-f175.google.com [209.85.213.175]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 9318C43BAC for ; Sun, 10 Jan 2016 23:05:40 +0000 (UTC) Received: by mail-ig0-f175.google.com with SMTP id z14so108322004igp.0 for ; Sun, 10 Jan 2016 15:05:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=I1p19ZzWue9OVnCGKyH/a3I6p5/Rm/IlCxNsdtApIJo=; b=KZ+14IRMvkbMU4+nW1DvGBPQzqzClt1xrVyQMB07ip/1OlB+1IHQKOs9X+o8pRp/81 LA4a54Wfc8y0o6pPZraW2DPKbIVkl0qa6XxWzP2NjlFYIrYSNmfp/h74fpnaL3tl/cYQ EXwR7nXZ383CO4x4RLo7Uduc4/ygtH9a19ippow+C5h2WrONwGa2pEl2sB6J7peJYSQv 8oLbzqC+xt6QjTuqGEFEwY/ts3fsKBm4DiGzQ4nNAxbSv0SIo1cj2p+IyFCNmeiQn8dE p4gmR1eBAYSKtOTzqyUBBK7kAev1Ur6R3Z0AttjHbHz8KLIbyOjFLUfNyNGvJXlJqxrd zFSg== X-Received: by 10.50.111.169 with SMTP id ij9mr9500112igb.92.1452467134431; Sun, 10 Jan 2016 15:05:34 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.18.71 with HTTP; Sun, 10 Jan 2016 15:05:15 -0800 (PST) In-Reply-To: References: From: Gavin Yue Date: Sun, 10 Jan 2016 15:05:15 -0800 Message-ID: Subject: Re: how to quickly fs -cp dir with thousand files? To: Chris Nauroth Cc: sandeep vura , "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=047d7b414318988903052902db87 --047d7b414318988903052902db87 Content-Type: text/plain; charset=UTF-8 Yes. I need two different copy. And I tried Chris's solution, distcp indeed works. Thank you all On Sun, Jan 10, 2016 at 3:00 PM, Chris Nauroth wrote: > Yes, certainly, if you only need it in one spot, then -mv is a fast > metadata-only operation. I was under the impression that Gavin really > wanted to achieve 2 distinct copies. Perhaps I was mistaken. > > --Chris Nauroth > > From: sandeep vura > Date: Sunday, January 10, 2016 at 6:23 AM > To: Chris Nauroth > Cc: Gavin Yue , "user@hadoop.apache.org" < > user@hadoop.apache.org> > Subject: Re: how to quickly fs -cp dir with thousand files? > > Hi Chris, > > Instead of copying files . Use mv command . > > > - hadoop fs -mv /user/hadoop/file1 /user/hadoop/file2 > > > Sandeep.v > > > On Sat, Jan 9, 2016 at 9:55 AM, Chris Nauroth > wrote: > >> DistCp is capable of running large copies like this in distributed >> fashion, implemented as a MapReduce job. >> >> http://hadoop.apache.org/docs/r2.7.1/hadoop-distcp/DistCp.html >> >> A lot of the literature on DistCp talks about use cases for copying >> across different clusters, but it's also completely legitimate to run >> DistCp within the same cluster. >> >> --Chris Nauroth >> >> From: Gavin Yue >> Date: Friday, January 8, 2016 at 4:45 PM >> To: "user@hadoop.apache.org" >> Subject: how to quickly fs -cp dir with thousand files? >> >> I want to cp a dir with over 8000 files to another dir in the same hdfs. >> but the copy process is really slow since it is copying one by one. >> Is there a fast way to copy this using Java FileSystem or FileUtil api? >> >> Thanks. >> >> > --047d7b414318988903052902db87 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Yes. I need two different copy. And=C2=A0 I tried Chr= is's solution, distcp indeed works.=C2=A0
Thank you all
<= /div>

On Sun, Jan = 10, 2016 at 3:00 PM, Chris Nauroth <cnauroth@hortonworks.com>= ; wrote:
Yes, certainly, if you only need it in one spot, then -mv is a fast me= tadata-only operation.=C2=A0 I was under the impression that Gavin really w= anted to achieve 2 distinct copies.=C2=A0 Perhaps I was mistaken.

--Chris Nauroth<= /font>

From: sandeep vura <sandeepvura@gmail.com>= ;
Date: Sunday, January 10, 2016 at 6= :23 AM
To: Chris Nauroth <cnauroth@hortonworks.com>
Cc: Gavin Yue <yue.yuanyuan@gmail.com>, = "user@hado= op.apache.org" <user@hadoop.apache.org>
Subject: Re: how to quickly fs -cp = dir with thousand files?

Hi Chris,

Instead of copying files . Use mv command .

  • hadoop fs -mv /user/ha= doop/file1 /user/hadoop/file2

Sandeep.v


On Sat, Jan 9, 2016 at 9:55 AM, Chris Nauroth <cnauroth@= hortonworks.com> wrote:
DistCp is capable of running large copies like this in distributed fas= hion, implemented as a MapReduce job.


A lot of the literature on DistCp talks about use cases for copying ac= ross different clusters, but it's also completely legitimate to run Dis= tCp within the same cluster.

--Chris Nauroth<= /font>

From: Gavin Yue <yue.yuanyuan@gmail.com>=
Date: Friday, January 8, 2016 at 4:= 45 PM
To: "user@hadoop.apache.org" <user@hadoop.apache= .org>
Subject: how to quickly fs -cp dir = with thousand files?

I want to cp a dir with over 8000 files to another dir in the same hdf= s.=C2=A0 but the copy process is really slow since it is copying one by one= .
Is there a fast way to copy this using Java FileSystem or FileUtil api?
Thanks.

--047d7b414318988903052902db87--