hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: how to quickly fs -cp dir with thousand files?
Date Sun, 10 Jan 2016 23:00:12 GMT
Yes, certainly, if you only need it in one spot, then -mv is a fast metadata-only operation.
 I was under the impression that Gavin really wanted to achieve 2 distinct copies.  Perhaps
I was mistaken.

--Chris Nauroth

From: sandeep vura <sandeepvura@gmail.com<mailto:sandeepvura@gmail.com>>
Date: Sunday, January 10, 2016 at 6:23 AM
To: Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>
Cc: Gavin Yue <yue.yuanyuan@gmail.com<mailto:yue.yuanyuan@gmail.com>>, "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
Subject: Re: how to quickly fs -cp dir with thousand files?

Hi Chris,

Instead of copying files . Use mv command .

  *   hadoop fs -mv /user/hadoop/file1 /user/hadoop/file2


On Sat, Jan 9, 2016 at 9:55 AM, Chris Nauroth <cnauroth@hortonworks.com<mailto:cnauroth@hortonworks.com>>
DistCp is capable of running large copies like this in distributed fashion, implemented as
a MapReduce job.


A lot of the literature on DistCp talks about use cases for copying across different clusters,
but it's also completely legitimate to run DistCp within the same cluster.

--Chris Nauroth

From: Gavin Yue <yue.yuanyuan@gmail.com<mailto:yue.yuanyuan@gmail.com>>
Date: Friday, January 8, 2016 at 4:45 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: how to quickly fs -cp dir with thousand files?

I want to cp a dir with over 8000 files to another dir in the same hdfs.  but the copy process
is really slow since it is copying one by one.
Is there a fast way to copy this using Java FileSystem or FileUtil api?


View raw message