hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Welling <well...@psc.edu>
Subject Re: Hadoop over Lustre?
Date Fri, 29 Aug 2008 20:58:27 GMT
That seems to have done the trick!  I am now running Hadoop 0.18
straight out of Lustre, without an intervening HDFS.  The unusual things
about my hadoop-site.xml are:

  <description>The shared directory where MapReduce stores control

where /bessemer/welling is a directory on a mounted Lustre filesystem.
I then do 'bin/start-mapred.sh' (without starting dfs), and I can run
Hadoop programs normally.  I do have to specify full input and output
file paths- they don't seem to be relative to fs.default.name .  That's
not too troublesome, though.

Thanks very much!  

On Fri, 2008-08-29 at 10:52 -0700, Owen O'Malley wrote:
> Check the setting for mapred.system.dir. This needs to be a path that is on
> a distributed file system. In old versions of Hadoop, it had to be on the
> default file system, but that is no longer true. In recent versions, the
> system dir only needs to be configured on the JobTracker and it is passed to
> the TaskTrackers and clients.

View raw message