hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: How can I deploy 100 blocks onto 10 datanodes with each node have 10 blocks?
Date Tue, 20 Oct 2009 12:05:44 GMT

On Oct 19, 2009, at 11:13 PM, Huang Qian wrote:

> How can I creat 10 files on each datanode? I think I can only create  
> files
> on filesystem,
> Is there any method to assign a file to a datanode?
>

No, and you probably don't want to.  At such a small scale, I can't  
think of a benefit to having the cluster *exactly* balanced.  Hadoop  
is designed to scale large, and it's simply too inefficient to put  
such a mechanism in place.

If you want *approximately* the same number of blocks on each node,  
you can use the rebalancer.

Brian

PS - if you really think this is mission-critical (and if you do, I'd  
advise to re-check your assumptions), you may want to look at the  
custom block placement plugins in the upcoming 0.21.0 release.

> 2009/10/19 Jason Venner <jason.hadoop@gmail.com>
>
>> If you set your replication count to one and on each datanode,  
>> create 10
>> files, you will achieve the pattern you are trying for.
>>
>> By default when a file is created on a machine hosting a datanode,  
>> that
>> datanode will receive 1 replica of the file, and will be  
>> responsible for
>> sending the file data to the next replica if any.
>>
>>
>> On Thu, Oct 15, 2009 at 1:39 PM, Huang Qian <skyswind@gmail.com>  
>> wrote:
>>
>>> Hi everyone. I am working on a project with hadoop and now I come  
>>> across
>>> some problem. How can I deploy 100 files, with each file have one  
>>> block
>> by
>>> setting the blocksize and controling the file size, on to 10  
>>> datanode,
>> and
>>> make sure each datanode has 10 blocks. I know the file system can  
>>> deploy
>>> the
>>> blocks automaticly, but I want to make sure for the assigns files,  
>>> the
>>> files
>>> will be deployed well-proportioned. How can I make it by the  
>>> hadoop tool
>> or
>>> api?
>>>
>>> Huang Qian(黄骞)
>>> Institute of Remote Sensing and GIS,Peking University
>>> Phone: (86-10) 5276-3109
>>> Mobile: (86) 1590-126-8883
>>> Address:Rm.554,Building 1,ChangChunXinYuan,Peking
>>> Univ.,Beijing(100871),CHINA
>>>
>>
>>
>>
>> --
>> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
>> http://www.amazon.com/dp/1430219424?tag=jewlerymall
>> www.prohadoopbook.com a community for Hadoop Professionals
>>


Mime
View raw message