hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: transfer large file onto hdfs
Date Fri, 06 Jul 2012 13:58:17 GMT
Thanks for the reply, Harsh.

BTW Does anyone deploy cdh3u4 on Solaris ?

On Thu, Jul 5, 2012 at 11:47 PM, Harsh J <harsh@cloudera.com> wrote:

> Should we assume that the non-hadoop system has no way to get on the
> network of the hadoop cluster and its clients? Otherwise, you can
> grant it temporary access and do a write from it itself. If not:
>
> Run a micro FTP server pointed to that file, and then do a 'hadoop fs
> -cp ftp://location hdfs://location', since FTPFileSystem is present in
> Hadoop? Or if NFS/etc. mounted, file:/// will work (or via
> copyFromLocal/put). Essentially you're bringing in the file remotely
> but performing the copy via CLI.
>
> Or you can copy them in chunks, either keeping the destination file
> writer open, if possible, or appending (depending on what version of
> Hadoop you're using).
>
> On Thu, Jul 5, 2012 at 11:54 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > Hi,
> > One of the customers wants to transfer dump file (size ~ 2TB) from
> outside
> > hadoop cluster onto hdfs.
> > The size exceeds free space on CLI machine.
> >
> > I want to poll best practice in this scenario.
> >
> > Thanks
>
>
>
> --
> Harsh J
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message