hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Macdonald <cra...@dcs.gla.ac.uk>
Subject Re: Hadoop+s3 & fuse-dfs
Date Thu, 29 Jan 2009 00:35:41 GMT
Hi Roopa,

Firstly, can you get the fuse-dfs working for an instance HDFS?
There is also a debug mode for fuse: enable this by adding -d on the 
command line.

C

Roopa Sudheendra wrote:
> Hey Craig,
>  I tried the way u suggested..but i get this transport endpoint not 
> connected. Can i see the logs anywhere? I dont see anything in 
> /var/log/messages either
>  looks like it tries to create the file system in hdfs.c but not sure 
> where it fails.
> I have the hadoop home set so i believe it gets the config info.
>
> any idea?
>
> Thanks,
> Roopa
> On Jan 28, 2009, at 1:59 PM, Craig Macdonald wrote:
>
>> In theory, yes.
>> On inspection of libhdfs, which underlies fuse-dfs, I note that:
>>
>> * libhdfs takes a host and port number as input when connecting, but 
>> not a scheme (hdfs etc). The easiest option would be to set the S3 as 
>> your default file system in your hadoop-site.xml, then use the host 
>> of "default". That should get libhdfs to use the S3 file system. i.e. 
>> set fuse-dfs to mount dfs://default:0/ and all should work as planned.
>>
>> * libhdfs also casts the FileSystem to a DistributedFileSystem for 
>> the df command. This would fail in your case. This issue is currently 
>> being worked on - see HADOOP-4368
>> https://issues.apache.org/jira/browse/HADOOP-4368.
>>
>> C
>>
>>
>> Roopa Sudheendra wrote:
>>> Thanks for the response craig.
>>> I looked at fuse-dfs c code and looks like it does not like anything 
>>> other than "dfs:// " so with the fact that hadoop can connect to S3 
>>> file system ..allowing s3 scheme should solve my problem?
>>>
>>> Roopa
>>>
>>> On Jan 28, 2009, at 1:03 PM, Craig Macdonald wrote:
>>>
>>>> Hi Roopa,
>>>>
>>>> I cant comment on the S3 specifics. However, fuse-dfs is based on a 
>>>> C interface called libhdfs which allows C programs (such as 
>>>> fuse-dfs) to connect to the Hadoop file system Java API. This being 
>>>> the case, fuse-dfs should (theoretically) be able to connect to any 
>>>> file system that Hadoop can. Your mileage may vary, but if you find 
>>>> issues, please do report them through the normal channels.
>>>>
>>>> Craig
>>>>
>>>>
>>>> Roopa Sudheendra wrote:
>>>>> I am experimenting with Hadoop backed by Amazon s3 filesystem as 
>>>>> one of our backup storage solution. Just the hadoop and s3(block 
>>>>> based since it overcomes the 5gb limit) so far seems to be fine.
>>>>> My problem is that i want to mount this filesystem using fuse-dfs 
>>>>> ( since i don't have to worry about how the file is written on the 
>>>>> system ) . Since the namenode does not get started with s3 backed 
>>>>> hadoop system how can i connect fuse-dfs to this setup.
>>>>>
>>>>> Appreciate your help.
>>>>> Thanks,
>>>>> Roopa
>>>>
>>>
>>
>


Mime
View raw message