hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakub Stransky <stransky...@gmail.com>
Subject Re: how to copy data between two hdfs cluster fastly?
Date Fri, 17 Oct 2014 19:53:41 GMT
Distcp?
On 17 Oct 2014 20:51, "Alexander Pivovarov" <apivovarov@gmail.com> wrote:

> try to run on dest cluster datanode
> $ hadoop fs -cp hdfs://from_cluster/....    hdfs://to_cluster/....
>
>
>
> On Fri, Oct 17, 2014 at 11:26 AM, Shivram Mani <smani@pivotal.io> wrote:
>
>> What is your approx input size ?
>> Do you have multiple files or is this one large file ?
>> What is your block size (source and destination cluster) ?
>>
>> On Fri, Oct 17, 2014 at 4:19 AM, ch huang <justlooks@gmail.com> wrote:
>>
>>> no ,all default
>>>
>>> On Fri, Oct 17, 2014 at 5:46 PM, Azuryy Yu <azuryyyu@gmail.com> wrote:
>>>
>>>> Did you specified how many map tasks?
>>>>
>>>>
>>>> On Fri, Oct 17, 2014 at 4:58 PM, ch huang <justlooks@gmail.com> wrote:
>>>>
>>>>> hi,maillist:
>>>>>              i now use distcp to migrate data from CDH4.4 to CDH5.1 ,
>>>>> i find when copy small file,it very good, but when transfer big data
,it
>>>>> very slow ,any good method recommand? thanks
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Thanks
>> Shivram
>>
>
>

Mime
View raw message