hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Albert Chern (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-3912) Improvements for NativeS3FileSystem
Date Wed, 06 Aug 2008 19:44:44 GMT
Improvements for NativeS3FileSystem

                 Key: HADOOP-3912
                 URL: https://issues.apache.org/jira/browse/HADOOP-3912
             Project: Hadoop Core
          Issue Type: Improvement
          Components: fs/s3
            Reporter: Albert Chern

In the process of porting NativeS3FileSystem for use with Hadoop 12, I made the following
changes and improvements which might be helpful (apologies if I should have opened separate
issues, but I was lazy):

1. The single byte read() method of NativeS3InputStream incorrectly treats the return value
of InputStream.read() as the number of bytes read, which is actually always 1.

2. It allows people to write files ending with the folder suffix.  I prevented this by doing
a check in the create() method.

3. Similarly, it allows people to open directories for reading.  I prevented this by doing
a check in the open() method.

4. If you write a file to a nonexistent directory tree, say /a/b/c/d/file, and then you delete/rename
that file or one of its parent directories, the whole directory tree vanishes.  I fixed this
by always creating the parent of a deleted/renamed file.

5. Recursive delete(), rename(), and getContentLength() can be sped up tremendously by working
directly with S3 listings rather than working at the FileSystem level.  All sub-files/sub-directories
should begin with the parent directory name as a prefix.

6. HADOOP-3506 is still relevant.

I don't have patches since I created a new file, but I have attached my source if Tom wants
to take a look at it.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message