hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: DistCP : Is it gauranteed to work for any two uri schemes?
Date Tue, 04 Feb 2014 13:33:15 GMT
Overall the whole DistCp utility is devoid of any HDFS specific items,
but does have some (mostly skippable) checks pertaining to FS level
features such as permissions, checksums, etc.. It should and does work
with any valid URI scheme that the libraries understand to be valid
FSes today.

On Tue, Feb 4, 2014 at 8:03 AM, Jay Vyas <jayunit100@gmail.com> wrote:
> Hi folks:
> I've been thinking about the AWS S3DistCP class and am wondering : is distcp
> built to work between any two hadoop file system classes ?
> Or is it implicitly built mainly to work to copy between to HDFS file
> sytems.
> I've havent found many examples online with different URI schemes.
> With emerging HDFS alternatives, I'd be interested in ways to otimize IO
> between different filesystems using distcp.
> --
> Jay Vyas
> http://jayunit100.blogspot.com

Harsh J

View raw message