hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <omal...@apache.org>
Subject Re: Integrating Lustre and HDFS
Date Thu, 10 Jun 2010 14:39:09 GMT
> Your local storage should get used for MR.  Use Lustre via file:// (LocalFileSystem, iirc)
> instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system

If Lustre has integrated checksums, you'll want to use the
RawLocalFileSystem instead of LocalFileSystem. You'll want to make it
accessible via:

fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem

so that urls like raw:///my/path won't go through the Hadoop checksum code.

-- Owen

View raw message