hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: adding new filesystems
Date Wed, 24 Mar 2010 16:52:34 GMT
On Wed, Mar 24, 2010 at 3:28 AM, Steve Loughran <stevel@apache.org> wrote:

>
> I'm looking at what it currently takes to implement new back end
> filestores, getting lost in the details.
>
> This is my current understanding -am I wrong?
>
> 1. There is an AbstractFileSystem, came in with HADOOP-6223, and is in
> SVN_HEAD only
> https://issues.apache.org/jira/browse/HADOOP-6223
>
>
And also FileContext.  AbstractFileSystem and FileContext are a refactoring
of FileSystem to separate the interfaces for file system developers (AFS)
and users of file systems (FC).

2. There is FileSystem, which exists and includes methods like append()
>
> 3. There is not currently any generic stress test for a filesystem other
> than terasort, but that doesn't really push the limit on FS operations,
> concurrency issues, appends, etc.
>

Not that I'm aware of.


>
> 4. There is not currently much in the way of subclasses of
> AbstractFileSystem other than a bridge to the classic FileSystem
>
>
The biggest is Hdfs (the replacement for DistributedFileSystem).


> 5. Because the AbstractFileSystem hasn't yet shipped, it offers the
> opportunity to add things like progress callbacks (HADOOP-6324).
>
>
That's my understanding. Adding new APIs to AFS is not out of the question.

6. Because the AbstractFileSystem hasn't yet shipped, if you want people on
> 0.20.x or 0.21.x to use your filesystem, you need to implement FileSystem.
>

Correct, at least for 20.  It's not clear if 21 will be released as is, or
skipped, or rebased on a future version of trunk (which will have AFS and
FC).

Thanks,
Eli

I'm curious as I need to know where to point some people. Short term,
> FileSystem looks like the one to target, and because AFS wraps it, you could
> stay with it for a while before worrying about AFS support altogether...
>
> -steve
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message