Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-Id: <9344919B-7180-402C-8A6C-732FD41CBB82@marketsystems.com>
From: Roopa Sudheendra <roopasudheendra@marketsystems.com>
To: core-user@hadoop.apache.org
In-Reply-To: <4980B937.8090009@dcs.gla.ac.uk>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: Hadoop+s3 & fuse-dfs
Date: Wed, 28 Jan 2009 17:45:38 -0600
References: <D6AB22A1-9678-415F-BA20-50187CE7AD6D@marketsystems.com>
 <4980ABEE.70805@dcs.gla.ac.uk>
 <3EF074F3-1F78-4363-AC77-F047F3F8F867@marketsystems.com>
 <4980B937.8090009@dcs.gla.ac.uk>

Hey Craig,
  I tried the way u suggested..but i get this transport endpoint not  
connected. Can i see the logs anywhere? I dont see anything in /var/ 
log/messages either
  looks like it tries to create the file system in hdfs.c but not sure  
where it fails.
I have the hadoop home set so i believe it gets the config info.

any idea?

Thanks,
Roopa
On Jan 28, 2009, at 1:59 PM, Craig Macdonald wrote:

> In theory, yes.
> On inspection of libhdfs, which underlies fuse-dfs, I note that:
>
> * libhdfs takes a host and port number as input when connecting, but  
> not a scheme (hdfs etc). The easiest option would be to set the S3  
> as your default file system in your hadoop-site.xml, then use the  
> host of "default". That should get libhdfs to use the S3 file  
> system. i.e. set fuse-dfs to mount dfs://default:0/ and all should  
> work as planned.
>
> * libhdfs also casts the FileSystem to a DistributedFileSystem for  
> the df command. This would fail in your case. This issue is  
> currently being worked on - see HADOOP-4368
> https://issues.apache.org/jira/browse/HADOOP-4368.
>
> C
>
>
> Roopa Sudheendra wrote:
>> Thanks for the response craig.
>> I looked at fuse-dfs c code and looks like it does not like  
>> anything other than "dfs:// " so with the fact that hadoop can  
>> connect to S3 file system ..allowing s3 scheme should solve my  
>> problem?
>>
>> Roopa
>>
>> On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:
>>
>>> Hi Roopa,
>>>
>>> I cant comment on the S3 specifics. However, fuse-dfs is based on  
>>> a C interface called libhdfs which allows C programs (such as fuse- 
>>> dfs) to connect to the Hadoop file system Java API. This being the  
>>> case, fuse-dfs should (theoretically) be able to connect to any  
>>> file system that Hadoop can. Your mileage may vary, but if you  
>>> find issues, please do report them through the normal channels.
>>>
>>> Craig
>>>
>>>
>>> Roopa Sudheendra wrote:
>>>> I am experimenting with Hadoop backed by Amazon s3 filesystem as  
>>>> one of our backup storage solution. Just the hadoop and s3(block  
>>>> based since it overcomes the 5gb limit) so far seems to be fine.
>>>> My problem is that i want to mount this filesystem using fuse-dfs  
>>>> ( since i don't have to worry about how the file is written on  
>>>> the system ) . Since the namenode does not get started with s3  
>>>> backed hadoop system how can i connect fuse-dfs to this setup.
>>>>
>>>> Appreciate your help.
>>>> Thanks,
>>>> Roopa
>>>
>>
>