hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikas Ashok Patil <vikas...@buffalo.edu>
Subject Re: Integrating Lustre and HDFS
Date Tue, 15 Jun 2010 10:31:09 GMT
Hello Allen,

Sorry for bugging you regarding the same problem again. If you say "we need
to be explicit having multiple file-systems" for map reduce jobs, are you
hinting on code changes to be made to hadoop ? Please provide more details
on this if possible.

Thanks,
Vikas

On Sat, Jun 12, 2010 at 9:05 AM, Vikas Ashok Patil <vikasash@buffalo.edu>wrote:

> Hello Allen,
>
> Thanks for the reply.
>
> You are right about trying to run two distributed filesystems. The reason
> being, there are certain restrictions (in our cluster environment) to
> include the local file system into lustre. Please tell me how would I make
> mapreduce access more than one file system. At least the configs don't seem
> to allow it.
>
> Thanks,
> Vikas A Patil
>
>
> On Sat, Jun 12, 2010 at 12:32 AM, Allen Wittenauer <
> awittenauer@linkedin.com> wrote:
>
>> On Jun 10, 2010, at 8:27 PM, Vikas Ashok Patil wrote:
>>
>> > Thanks for the replies.
>> >
>> > If I have fs.default.name = file://my_lustre_mount_point , then only
>> the
>> > lustre filesystem will be used. I would like to have something like
>> >
>> > fs.default.name=file://my_lustre_mount_point , hdfs://localhost:9123
>> >
>> > so that both local filesystem and lustre are in use.
>> >
>> > Kindly correct me if I am missing something here.
>>
>> I guess we're all confused as to your use case.  Why do you want to run
>> two distributed file systems on the same nodes?  Why can't you use Lustre
>> for all your needs?
>>
>> As to fs.default.name, you can only have one.  [That's why it is a
>> default. *smile*]  If you want to access more than one file system from
>> within MapReduce, you'll need to specify it explicitly.
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message