hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Stack <st...@archive.org>
Subject Re: s3
Date Sat, 13 Jan 2007 22:06:51 GMT
Tom White wrote:
>> Other things I notice are that the '-rmr'/'-lsr' options don't act as
>> 'expected'.  Its a little confusing.  Should the 'hadoop fs' tool return
>> from rmr/lsr rather be that the action is not supported rather than
>> 'success' if, say, I try to remove a 'directory' from S3?  (See below
>> for illustrative output).
> I've written a patch to fix this. Please see
> https://issues.apache.org/jira/browse/HADOOP-880.
Thanks for the above.

I also see you have put up a wiki page on using S3 that stalls out at 
HADOOP-862, the adding of s3 support to CopyFiles.  I can complete the 
page after/if HADOOP-862 is applied.

> I think it would be relatively easy to add support for reading regular
> S3 files in the current implementation. Doing this with S3 metadata
> rather than file "magic" seems right to me. There are two things we
> would put in the metadata of files written by S3FileSystem: an
> indication that it is block oriented ("S3FileSystem.type=block") and a
> filesystem version number ("S3FileSystem.version=1.0"). Regular S3
> files would not have the type metadata so S3FileSystem would not try
> to interpret them as inodes.
> ...
And +1 on this development of the Bryan Pendleton metadata suggestion 
for distinguishing between files-are-lists-of-blocks vs. actual files up 
in S3 (Much cleaner than my distinguishing by file magic).


View raw message