hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan A. P. Pendleton" ...@geekdom.net>
Subject Re: s3
Date Sat, 13 Jan 2007 22:27:49 GMT
Not sure what's wrong, but I've been continuously running into problems with
S3 for the last day or so trying to use the  hadoop s3 support with it. The
persistent extension is "A conflicting conditional operation is currently in
progress against this resource. Please try again.", which seems related to
the problem being discussed with a different toolkit on the s3 message
board:
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=13533&tstart=0(the
problem in the hadoop s3 implementation originates from
org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(
Jets3tFileSystemStore.java:85)


On 1/13/07, Michael Stack <stack@archive.org> wrote:
>
> Tom White wrote:
> >> Other things I notice are that the '-rmr'/'-lsr' options don't act as
> >> 'expected'.  Its a little confusing.  Should the 'hadoop fs' tool
> return
> >> from rmr/lsr rather be that the action is not supported rather than
> >> 'success' if, say, I try to remove a 'directory' from S3?  (See below
> >> for illustrative output).
> >
> > I've written a patch to fix this. Please see
> > https://issues.apache.org/jira/browse/HADOOP-880.
> >
> Thanks for the above.
>
> I also see you have put up a wiki page on using S3 that stalls out at
> HADOOP-862, the adding of s3 support to CopyFiles.  I can complete the
> page after/if HADOOP-862 is applied.
>
> > I think it would be relatively easy to add support for reading regular
> > S3 files in the current implementation. Doing this with S3 metadata
> > rather than file "magic" seems right to me. There are two things we
> > would put in the metadata of files written by S3FileSystem: an
> > indication that it is block oriented ("S3FileSystem.type=block") and a
> > filesystem version number ("S3FileSystem.version=1.0"). Regular S3
> > files would not have the type metadata so S3FileSystem would not try
> > to interpret them as inodes.
> > ...
> And +1 on this development of the Bryan Pendleton metadata suggestion
> for distinguishing between files-are-lists-of-blocks vs. actual files up
> in S3 (Much cleaner than my distinguishing by file magic).
>
> St.Ack
>



-- 
Bryan A. P. Pendleton
Ph: (877) geek-1-bp

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message