hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: Integrating Lustre and HDFS
Date Thu, 10 Jun 2010 13:57:31 GMT

On Jun 10, 2010, at 2:37 AM, Vikas Ashok Patil wrote:
> In my cluster I have the lustre parallel file system which mainly exposes
> storage over a network. Also there is some local space on each node of the
> cluster. This space is not part of the lustre file system. My hadoop
> installation currently makes use of this local file system. However, I would
> like to make use of the space available through lustre(exposed over the
> network). Hence I was thinking of a way to integrate HDFS and lustre, where
> HDFS would manage the local storage and lustre would provide the storage
> over the network.
> Please let me know your thoughts on this.

Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc)
instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.
View raw message