hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: transfer large file onto hdfs
Date Fri, 06 Jul 2012 06:47:06 GMT
Should we assume that the non-hadoop system has no way to get on the
network of the hadoop cluster and its clients? Otherwise, you can
grant it temporary access and do a write from it itself. If not:

Run a micro FTP server pointed to that file, and then do a 'hadoop fs
-cp ftp://location hdfs://location', since FTPFileSystem is present in
Hadoop? Or if NFS/etc. mounted, file:/// will work (or via
copyFromLocal/put). Essentially you're bringing in the file remotely
but performing the copy via CLI.

Or you can copy them in chunks, either keeping the destination file
writer open, if possible, or appending (depending on what version of
Hadoop you're using).

On Thu, Jul 5, 2012 at 11:54 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> Hi,
> One of the customers wants to transfer dump file (size ~ 2TB) from outside
> hadoop cluster onto hdfs.
> The size exceeds free space on CLI machine.
> I want to poll best practice in this scenario.
> Thanks

Harsh J

View raw message