hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Will Maier <wcma...@hep.wisc.edu>
Subject Re: hdfs file - datanode association
Date Thu, 06 Oct 2011 13:12:12 GMT
Hi Vincent-

On Thu, Oct 06, 2011 at 11:19:20AM +0200, Vincent Boucher wrote:
> We are wondering if it is possible to require that the namenode redirects the 
> blocks of the files of a given directory to a particular set of datanodes?
> Our case is the following:
>  - Servers
>     10 x mass storage servers of each 50TB, RAID6
>        -> 500TB available for hdfs
>     30 x working nodes with 6TB (no RAID)
>        -> 180TB available for hdfs
> We'd like that the files stored in
>    /hdfs/ms
> be hosted on the mass storage (ms) servers,
> while the files in
>   /hdfs/wn
> be hosted on the working nodes (wn).

Why do you want to partition your datanodes this way? In our cluster, datanode
size ranges from ~1 TB (no RAID) up to ~80TB (RAID). While in an ideal world all
datanodes would be similar in configuration, we have not observed any issues
with this arrangement in production.


Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/

View raw message