hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Hadoop over Lustre?
Date Thu, 21 Aug 2008 18:13:03 GMT
It wouldn't be too much of a stretch to use Lustre directly...  
although it isn't trivial either.
You'd need to implement the 'FileSystem' interface for Lustre, define  
a URI scheme (e.g. lfs://) etc. Please take a take a look at the KFS/ 
S3 implementations.


On Aug 21, 2008, at 9:59 AM, Joel Welling wrote:

> Hi folks;
>  I'm new to Hadoop, and I'm trying to set it up on a cluster for which
> almost all the disk is mounted via the Lustre filesystem.  That
> filesystem is visible to all the nodes, so I don't actually need  
> HDFS to
> implement a shared filesystem.  (I know the philosophical reasons why
> people say local disks are better for Hadoop, but that's not the
> situation I've got).  My system is failing, and I think it's because  
> the
> different nodes are tripping over each other when they try to run HDFS
> out of the same directory tree.
>  Is there a way to turn off HDFS and just let Lustre do the  
> distributed
> filesystem?  I've seen discussion threads about Hadoop with NFS which
> said something like 'just specify a local filesystem and everything  
> will
> be fine', but I don't know how to do that.  I'm using Hadoop 0.17.2.
> Thanks, I hope;
> -Joel Welling
> welling@psc.edu

View raw message