hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom White <...@cloudera.com>
Subject Re: adding new filesystems
Date Wed, 24 Mar 2010 17:18:26 GMT
Steve,

For testing, have a look at FileSystemContractBaseTest and the
FileContext*BaseTest classes. You can subclass these to get a lot of
basic tests for free. (These don't cover the kind of stress tests you
mentioned though.)

Cheers
Tom

On Wed, Mar 24, 2010 at 3:28 AM, Steve Loughran <stevel@apache.org> wrote:
>
> I'm looking at what it currently takes to implement new back end filestores,
> getting lost in the details.
>
> This is my current understanding -am I wrong?
>
> 1. There is an AbstractFileSystem, came in with HADOOP-6223, and is in
> SVN_HEAD only
> https://issues.apache.org/jira/browse/HADOOP-6223
>
> 2. There is FileSystem, which exists and includes methods like append()
>
> 3. There is not currently any generic stress test for a filesystem other
> than terasort, but that doesn't really push the limit on FS operations,
> concurrency issues, appends, etc.
>
> 4. There is not currently much in the way of subclasses of
> AbstractFileSystem other than a bridge to the classic FileSystem
>
> 5. Because the AbstractFileSystem hasn't yet shipped, it offers the
> opportunity to add things like progress callbacks (HADOOP-6324).
>
> 6. Because the AbstractFileSystem hasn't yet shipped, if you want people on
> 0.20.x or 0.21.x to use your filesystem, you need to implement FileSystem.
>
> I'm curious as I need to know where to point some people. Short term,
> FileSystem looks like the one to target, and because AFS wraps it, you could
> stay with it for a while before worrying about AFS support altogether...
>
> -steve
>

Mime
View raw message