hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Soztutar <enis.soz.nu...@gmail.com>
Subject Re: hadoop file system browser
Date Thu, 24 Jan 2008 10:31:32 GMT
Yes, you can solve the bottleneck by starting a webdav server on each 
client. But this would include the burden to manage the servers etc. and 
it may not be the intended use case for webdav. But we can further 
discuss the architecture in the relevant issue.

Alban Chevignard wrote:
> Thanks for the clarification. I agree that running a single WebDAV
> server for all clients would make it a bottleneck. But I can't see
> anything in the current WebDAV server implementation that precludes
> running an instance of it on each client. It seems to me that would
> solve any bottleneck issue.
>
> -Alban
>
> On Jan 23, 2008 2:53 AM, Enis Soztutar <enis.soz.nutch@gmail.com> wrote:
>   
>> As you know, dfs client connects to the individual datanodes to
>> read/write data and has a minimal interaction with the Namenode, which
>> improves the io rate linearly(theoretically 1:1). However current
>> implementation of webdav interface, is just a server working on a single
>> machine, which translates the webdav requests to namenode. Thus the
>> whole traffic passes through this webdav server, which makes it a
>> bottleneck. I was planning to integrate webdav server with
>> namenode/datanode, and forward the requests to the other datanodes, so
>> that we can do io in parallel, but my focus on webdav has faded for now.
>>
>>
>>
>>
>> Alban Chevignard wrote:
>>     
>>> What are the scalability issues associated with the current WebDAV interface?
>>>
>>> Thanks,
>>> -Alban
>>>
>>> On Jan 22, 2008 7:27 AM, Enis Soztutar <enis.soz.nutch@gmail.com> wrote:
>>>
>>>       
>>>> Webdav interface for hadoop works as it is, but it needs a major
>>>> redesign to be scalable, however it is still useful. It has even been
>>>> used with windows explorer defining the webdav server as a remote service.
>>>>
>>>>
>>>> Ted Dunning wrote:
>>>>
>>>>         
>>>>> There has been significant work on building a web-DAV interface for HDFS.
 I
>>>>> haven't heard any news for some time, however.
>>>>>
>>>>>
>>>>> On 1/21/08 11:32 AM, "Dawid Weiss" <dawid.weiss@cs.put.poznan.pl>
wrote:
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>>>> The Eclipse plug-in also features a DFS browser.
>>>>>>>
>>>>>>>
>>>>>>>               
>>>>>> Yep. That's all true, I don't mean to self-promote, because there
really isn't
>>>>>> that much to advertise ;) I was just quite attached to file manager-like
user
>>>>>> interface; the mucommander clone I posted served me as a browser,
but also for
>>>>>> rudimentary file operations (copying to/from, deleting folders etc.).
In my
>>>>>> experience it's been quite handy.
>>>>>>
>>>>>> It would be probably a good idea to implement a commons-vfs plugin
for Hadoop
>>>>>> so
>>>>>> that HDFS filesystem is transparent to use for other apps.
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>>
>>>>>>             
>>>>>           
>>>       
>
>   

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message