hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SF Hadoop <sfhad...@gmail.com>
Subject Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Date Sat, 20 Jun 2015 06:45:39 GMT
Really depends on your requirements for the format of the data.

The easiest way I can think of is to "stream" batches of data into a pub
sub system that the target system can access and then consume.

Verify each batch and then ditch them.

You can throttle the size of the intermediary infrastructure based on your

Seems the most efficient approach.

On Thursday, June 18, 2015, Divya Gehlot <divya.htconex@gmail.com> wrote:

> Hi,
> I need to copy data from first hadoop cluster to second hadoop cluster.
> I cant access second hadoop cluster from first hadoop cluster due to some
> security issue.
> Can any point me how can I do apart from distcp command.
> For instance
> Cluster 1 secured zone -> copy hdfs data  to -> cluster 2 in non secured
> zone
> Thanks,
> Divya

View raw message