hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Soztutar <enis.soz.nu...@gmail.com>
Subject Re: hadoop file system browser
Date Wed, 23 Jan 2008 07:53:22 GMT
As you know, dfs client connects to the individual datanodes to 
read/write data and has a minimal interaction with the Namenode, which 
improves the io rate linearly(theoretically 1:1). However current 
implementation of webdav interface, is just a server working on a single 
machine, which translates the webdav requests to namenode. Thus the 
whole traffic passes through this webdav server, which makes it a 
bottleneck. I was planning to integrate webdav server with 
namenode/datanode, and forward the requests to the other datanodes, so 
that we can do io in parallel, but my focus on webdav has faded for now.



Alban Chevignard wrote:
> What are the scalability issues associated with the current WebDAV interface?
>
> Thanks,
> -Alban
>
> On Jan 22, 2008 7:27 AM, Enis Soztutar <enis.soz.nutch@gmail.com> wrote:
>   
>> Webdav interface for hadoop works as it is, but it needs a major
>> redesign to be scalable, however it is still useful. It has even been
>> used with windows explorer defining the webdav server as a remote service.
>>
>>
>> Ted Dunning wrote:
>>     
>>> There has been significant work on building a web-DAV interface for HDFS.  I
>>> haven't heard any news for some time, however.
>>>
>>>
>>> On 1/21/08 11:32 AM, "Dawid Weiss" <dawid.weiss@cs.put.poznan.pl> wrote:
>>>
>>>
>>>       
>>>>> The Eclipse plug-in also features a DFS browser.
>>>>>
>>>>>           
>>>> Yep. That's all true, I don't mean to self-promote, because there really
isn't
>>>> that much to advertise ;) I was just quite attached to file manager-like
user
>>>> interface; the mucommander clone I posted served me as a browser, but also
for
>>>> rudimentary file operations (copying to/from, deleting folders etc.). In
my
>>>> experience it's been quite handy.
>>>>
>>>> It would be probably a good idea to implement a commons-vfs plugin for Hadoop
>>>> so
>>>> that HDFS filesystem is transparent to use for other apps.
>>>>
>>>> Dawid
>>>>
>>>>         
>>>
>>>       
>
>   

Mime
View raw message