hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject adding new filesystems
Date Wed, 24 Mar 2010 10:28:51 GMT

I'm looking at what it currently takes to implement new back end 
filestores, getting lost in the details.

This is my current understanding -am I wrong?

1. There is an AbstractFileSystem, came in with HADOOP-6223, and is in 

2. There is FileSystem, which exists and includes methods like append()

3. There is not currently any generic stress test for a filesystem other 
than terasort, but that doesn't really push the limit on FS operations, 
concurrency issues, appends, etc.

4. There is not currently much in the way of subclasses of 
AbstractFileSystem other than a bridge to the classic FileSystem

5. Because the AbstractFileSystem hasn't yet shipped, it offers the 
opportunity to add things like progress callbacks (HADOOP-6324).

6. Because the AbstractFileSystem hasn't yet shipped, if you want people 
on 0.20.x or 0.21.x to use your filesystem, you need to implement 

I'm curious as I need to know where to point some people. Short term, 
FileSystem looks like the one to target, and because AFS wraps it, you 
could stay with it for a while before worrying about AFS support 


View raw message