hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: how to quickly fs -cp dir with thousand files?
Date Sat, 09 Jan 2016 04:25:16 GMT
DistCp is capable of running large copies like this in distributed fashion, implemented as
a MapReduce job.


A lot of the literature on DistCp talks about use cases for copying across different clusters,
but it's also completely legitimate to run DistCp within the same cluster.

--Chris Nauroth

From: Gavin Yue <yue.yuanyuan@gmail.com<mailto:yue.yuanyuan@gmail.com>>
Date: Friday, January 8, 2016 at 4:45 PM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: how to quickly fs -cp dir with thousand files?

I want to cp a dir with over 8000 files to another dir in the same hdfs.  but the copy process
is really slow since it is copying one by one.
Is there a fast way to copy this using Java FileSystem or FileUtil api?


View raw message