hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma" <jssa...@facebook.com>
Subject RE: Question on DFS block placement and 'what is a rack' wrt DFS block placement
Date Tue, 12 Feb 2008 23:53:44 GMT
There may still be remaining issues with. One I am aware of is
https://issues.apache.org/jira/browse/HADOOP-2677 where smaller capacity
nodes become too highly utilized to store mapred intermediate output.


-----Original Message-----
From: Jason Venner [mailto:jason@attributor.com] 
Sent: Tuesday, February 12, 2008 12:02 PM
To: core-user@hadoop.apache.org
Subject: Re: Question on DFS block placement and 'what is a rack' wrt
DFS block placement

We are currently running 15.3, and hope to move to 16.1 when it comes
out...
Where the heterogeneous disk space issues fixed in15.3?

Ted Dunning wrote:
> I have had issues with machines that are highly disparate in terms of
disk
> space.  I expect that some of those issues have been mitigated in
recent
> releases.
>
>
> On 2/12/08 11:51 AM, "Jason Venner" <jason@attributor.com> wrote:
>
>   
>> We are starting to build larger clusters, and want to better
understand
>> how to configure the network topology.
>> Up to now we have just been setting up a private vlan for the small
>> clusters.
>>
>> We have been thinking about the following machine configurations
>> Compute nodes with a number of spindles and medium disk, that also
serve DFS
>> For every 4-8 of the above, one compute node with a large number of
>> spindles with a large number of disks, to bulk out th DFS capacity.
>>
>> We are wondering what the best practices are for network topology in
>> clusters that are built out of the above building blocks.
>> We can readily have 2 or 4 network cards in each node.
>>     
>
>   

Mime
View raw message