hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Hadoop over Lustre?
Date Fri, 22 Aug 2008 14:48:55 GMT
Joel Welling wrote:
> Thanks, Steve and Arun.  I'll definitely try to write something based on
> the KFS interface.  I think that for our applications putting the mapper
> on the right rack is not going to be that useful.  A lot of our
> calculations are going to be disordered stuff based on 3D spatial
> relationships like nearest-neighbor finding, so things will be in a
> random access pattern most of the time.
> 
> Is there a way to set up the configuration for HDFS so that different
> datanodes keep their data in different directories?  That would be a big
> help in the short term.

yes, but you'd have to push out a different config to each datanode.

1. I have some stuff that could help there, but its not ready for 
production use yet [1].

2. Could you set up symlinks from the local filesystem, so point every 
node at a local dir
  /tmp/hadoop
with each node pointing to a different subdir in the big filesystem?


[1] 
http://people.apache.org/~stevel/slides/deploying_hadoop_with_smartfrog.pdf

Mime
View raw message