hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: using distcp for http source files
Date Thu, 22 Jan 2009 19:09:39 GMT
Doesn't the WebDAV protocol use http for file transfer, and support reads /
writes / listings / etc? Is anyone aware of an OSS web dav library that
could be wrapped in a FileSystem implementation?

- Aaron

On Wed, Jan 21, 2009 at 3:30 PM, Doug Cutting <cutting@apache.org> wrote:

> Derek Young wrote:
>
>> Reading http://issues.apache.org/jira/browse/HADOOP-341 it sounds like
>> this should be supported, but the http URLs are not working for me.  Are
>> http source URLs still supported?
>>
>
> No.  They used to be supported, but when distcp was converted to accept any
> Path this stopped working, since there is no FileSystem implementation
> mapped to http: paths.  Implementing an HttpFileSystem that supports
> read-only access to files and no directory listings is fairly trivial, but
> without directory listings, distcp would not work well.
>
> https://issues.apache.org/jira/browse/HADOOP-1563 includes a now
> long-stale patch that implements an HTTP filesystem, where directory
> listings are implemented, assuming that:
>  - directories are represented by slash-terminated urls;
>  - GET of a directory contains the URLs of its children
> This works for the directory listings returned by many HTTP servers.
>
> Perhaps someone can update this patch, and, if folks find it useful, we can
> include it.
>
> Doug
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message