hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhuweimin" <xim-...@tsm.kddilabs.jp>
Subject RE: Copying files between two remote hadoop clusters
Date Mon, 08 Mar 2010 00:55:14 GMT
Hi 

Hdfs shell command standard I/O is supported. I think if you using it for
avoiding temporary save to local file system.

For example:
wget https://web-server/file-path -O - | hadoop fs -put -
hdfs://nn.example.com/hadoop/hadoopfile

Refer to this URL
http://hadoop.apache.org/common/docs/current/hdfs_shell.html#put

Hope that helps

zhuweimin

> -----Original Message-----
> From: zenMonkey [mailto:numan.salati@gmail.com]
> Sent: Sunday, March 07, 2010 4:23 AM
> To: hadoop-user@lucene.apache.org
> Subject: Copying files between two remote hadoop clusters
> 
> 
> I want to write a script that pulls data (flat files) from a remote
machine
> and pushes that into its hadoop cluster.
> 
> At the moment, it is done in two steps:
> 
> 1 - Secure copy the remote files
> 2 - Put the files into HDFS
> 
> I was wondering if it was possible to optimize this by avoiding copying
> to
> local fs before pushing to hdfs; and instead write directly to hdfs. I am
> not sure if this is something that hadoop tools already provide.
> 
> Thanks for any help.
> 
> --
> View this message in context:
> http://old.nabble.com/Copying-files-between-two-remote-hadoop-clusters
> -tp27799963p27799963.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.




Mime
View raw message