hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stanley Shi <s...@gopivotal.com>
Subject Re: Managed File Transfer
Date Wed, 09 Jul 2014 08:39:46 GMT
There's a DistCP utility for this kind of purpose;
Also there's "Spring XD" there, but I am not sure if you want to use it.

*Stanley Shi,*

On Mon, Jul 7, 2014 at 10:02 PM, Mohan Radhakrishnan <
radhakrishnan.mohan@gmail.com> wrote:

> Hi,
>            We used a commercial FT and scheduler tool in clustered mode.
> This was a traditional active-active cluster that supported multiple
> protocols like FTPS etc.
>     Now I am interested in evaluating a Distributed way of crawling FTP
> sites and downloading files using Hadoop. I thought since we have to
> process thousands of files Hadoop jobs can do it.
> Are Hadoop jobs used for this type of file transfers ?
> Moreover there is a requirement for a scheduler  also. What is the
> recommendation of the forum ?
> Thanks,
> Mohan

View raw message