hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zenMonkey <numan.sal...@gmail.com>
Subject Copying files between two remote hadoop clusters
Date Sat, 06 Mar 2010 19:23:11 GMT

I want to write a script that pulls data (flat files) from a remote machine
and pushes that into its hadoop cluster.

At the moment, it is done in two steps:

1 - Secure copy the remote files
2 - Put the files into HDFS

I was wondering if it was possible to optimize this by avoiding copying to
local fs before pushing to hdfs; and instead write directly to hdfs. I am
not sure if this is something that hadoop tools already provide. 

Thanks for any help.

View this message in context: http://old.nabble.com/Copying-files-between-two-remote-hadoop-clusters-tp27799963p27799963.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

View raw message