Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1392781043.1212523485322.JavaMail.jira@brutus>
Date: Tue, 3 Jun 2008 13:04:45 -0700 (PDT)
From: "Tom White (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-930) Add support for reading regular
 (non-block-based) files from S3 in S3FileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602056#action_12602056 ] 

Tom White commented on HADOOP-930:
----------------------------------

Thanks for the review Chris.

bq. Any reason you didn't use the mime type to denote directory files (as jets3t does)?

It's to do with efficiency of listing directories. If you use mime type then you can't tell the difference between files and directories when listing bucket keys. So you have to query each key in a directory which can be prohibitively slow. But if you use the _$folder$ suffix convention (which S3Fox uses too BTW) you can easily distinguish files and directories. 

bq. I believe MD5 checksum should be set on s3 put (via header), and verified on s3 get. 

The code should be doing this. I agree that it's useful - in fact, the other s3 filesystem needs updating to do this too.

bq. Sometimes 'legacy' buckets have underscores, might consider trying to survive them.

Thanks for the tip. The code does detect this condition, but it might be nice to try to workaround as you say (perhaps emitting a warning). Have you done this elsewhere?

> Add support for reading regular (non-block-based) files from S3 in S3FileSystem
> -------------------------------------------------------------------------------
>
>                 Key: HADOOP-930
>                 URL: https://issues.apache.org/jira/browse/HADOOP-930
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 0.10.1
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.18.0
>
>         Attachments: hadoop-930-v2.patch, hadoop-930-v3.patch, hadoop-930-v4.patch, hadoop-930.patch, jets3t-0.6.0.jar
>
>
> People often have input data on S3 that they want to use for a Map Reduce job and the current S3FileSystem implementation cannot read it since it assumes a block-based format.
> We would add the following metadata to files written by S3FileSystem: an indication that it is block oriented ("S3FileSystem.type=block") and a filesystem version number ("S3FileSystem.version=1.0"). Regular S3 files would not have the type metadata so S3FileSystem would not try to interpret them as inodes.
> An extension to write regular files to S3 would not be covered by this change - we could do this as a separate piece of work (we still need to decide whether to introduce another scheme - e.g. rename block-based S3 to "s3fs" and call regular S3 "s3" - or whether to just use a configuration property to control block-based vs. regular writes).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.