hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From max scalf <oracle.bl...@gmail.com>
Subject Re: copy data from one hadoop cluster to another hadoop cluster + cant use distcp
Date Sat, 20 Jun 2015 00:44:18 GMT
Not to hijack this post but how would you deal with data that is maintained
by hive(Orc format file, hive created tables etc..)...Would we copy the
hivemetastore(MySQL) and move that over to new cluster?

On Friday, June 19, 2015, Joep Rottinghuis <jrottinghuis@gmail.com> wrote:

> You can't set up a proxy ?
> You probably want to avoid writing to local file system because aside from
> that being slow, it limits the size of your file to the free space on your
> local disc.
>
> If you do need to go commando and go through a single client machine that
> can see both clusters you probably want to pipe a get to a put.
>
> Any kind of serious data volume pulled through a "straw" is going to be
> rather slow though.
>
> Cheers,
>
> Joep
>
> Sent from my iPhone
>
> On Jun 19, 2015, at 12:09 AM, Nitin Pawar <nitinpawar432@gmail.com
> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>
> yes
>
> On Fri, Jun 19, 2015 at 11:36 AM, Divya Gehlot <divya.htconex@gmail.com
> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>
>> In thats It will be like three step process .
>> 1. first cluster (secure zone) HDFS  -> copytoLocal -> user local file
>> system
>> 2. user local space -> copy data -> second cluster user local file system
>> 3. second cluster user local file system -> copyfromlocal -> second
>> clusterHDFS
>>
>> Am I on the right track ?
>>
>>
>>
>> On 19 June 2015 at 12:38, Nitin Pawar <nitinpawar432@gmail.com
>> <javascript:_e(%7B%7D,'cvml','nitinpawar432@gmail.com');>> wrote:
>>
>>> What's the size of the data?
>>> If you can not do distcp between clusters then other way is doing hdfs
>>> get on the data and then hdfs put on another cluster
>>> On 19-Jun-2015 9:56 am, "Divya Gehlot" <divya.htconex@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','divya.htconex@gmail.com');>> wrote:
>>>
>>>> Hi,
>>>> I need to copy data from first hadoop cluster to second hadoop cluster.
>>>> I cant access second hadoop cluster from first hadoop cluster due to
>>>> some security issue.
>>>> Can any point me how can I do apart from distcp command.
>>>> For instance
>>>> Cluster 1 secured zone -> copy hdfs data  to -> cluster 2 in non
>>>> secured zone
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>>
>>>>
>>
>
>
> --
> Nitin Pawar
>
>

Mime
View raw message