hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4422) S3 file systems should not create bucket
Date Tue, 25 Nov 2008 18:39:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650674#action_12650674

Hudson commented on HADOOP-4422:

Integrated in Hadoop-trunk #670 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/670/])
    . S3 file systems should not create bucket. Contributed by David Phillips.

> S3 file systems should not create bucket
> ----------------------------------------
>                 Key: HADOOP-4422
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4422
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.18.1
>            Reporter: David Phillips
>            Assignee: David Phillips
>             Fix For: 0.20.0
>         Attachments: hadoop-s3n-nocreate.patch, hadoop-s3n-nocreate.patch
> Both S3 file systems (s3 and s3n) try to create the bucket at every initialization. 
This is bad because
> * Every S3 operation costs money.  These unnecessary calls are an unnecessary expense.
> * These calls can fail when called concurrently.  This makes the file system unusable
in large jobs.
> * Any operation, such as a "fs -ls", creates a bucket.  This is counter-intuitive and
> The initialization code should assume the bucket exists:
> * Creating a bucket is a very rare operation.  Accounts are limited to 100 buckets.
> * Any check at initialization for bucket existence is a waste of money.
> Per Amazon: "Because bucket operations work against a centralized, global resource space,
it is not appropriate to make bucket create or delete calls on the high availability code
path of your application. It is better to create or delete buckets in a separate initialization
or setup routine that you run less often."

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message