hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joep Rottinghuis <jrottingh...@gmail.com>
Subject Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Date Fri, 19 Jun 2015 08:31:27 GMT
You can't set up a proxy ?
You probably want to avoid writing to local file system because aside from that being slow,
it limits the size of your file to the free space on your local disc.

If you do need to go commando and go through a single client machine that can see both clusters
you probably want to pipe a get to a put.

Any kind of serious data volume pulled through a "straw" is going to be rather slow though.

Cheers,

Joep

Sent from my iPhone

> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <nitinpawar432@gmail.com> wrote:
> 
> yes 
> 
>> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <divya.htconex@gmail.com> wrote:
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS  -> copytoLocal -> user local file system

>> 2. user local space -> copy data -> second cluster user local file system 
>> 3. second cluster user local file system -> copyfromlocal -> second clusterHDFS

>> 
>> Am I on the right track ?
>> 
>>  
>> 
>>> On 19 June 2015 at 12:38, Nitin Pawar <nitinpawar432@gmail.com> wrote:
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs get on
the data and then hdfs put on another cluster
>>> 
>>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <divya.htconex@gmail.com> wrote:
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to some
security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance 
>>>> Cluster 1 secured zone -> copy hdfs data  to -> cluster 2 in non secured
zone 
>>>>  
>>>> 
>>>> 
>>>> Thanks,
>>>> Divya
> 
> 
> 
> -- 
> Nitin Pawar

Mime
View raw message