hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikas Ashok Patil <vikas...@buffalo.edu>
Subject Re: Integrating Lustre and HDFS
Date Fri, 11 Jun 2010 03:27:46 GMT
Thanks for the replies.

If I have fs.default.name = file://my_lustre_mount_point , then only the
lustre filesystem will be used. I would like to have something like

fs.default.name=file://my_lustre_mount_point , hdfs://localhost:9123

so that both local filesystem and lustre are in use.

Kindly correct me if I am missing something here.

Thanks,
Vikas

On Thu, Jun 10, 2010 at 8:09 PM, Owen O'Malley <omalley@apache.org> wrote:

> > Your local storage should get used for MR.  Use Lustre via file://
> (LocalFileSystem, iirc)
> > instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default
> file system type.
>
> If Lustre has integrated checksums, you'll want to use the
> RawLocalFileSystem instead of LocalFileSystem. You'll want to make it
> accessible via:
>
> fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem
>
> so that urls like raw:///my/path won't go through the Hadoop checksum code.
>
> -- Owen
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message