hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Demoor (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-11183) Memory-based S3AOutputstream
Date Thu, 09 Oct 2014 15:42:33 GMT
Thomas Demoor created HADOOP-11183:

             Summary: Memory-based S3AOutputstream
                 Key: HADOOP-11183
                 URL: https://issues.apache.org/jira/browse/HADOOP-11183
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/s3
    Affects Versions: 2.4.0
            Reporter: Thomas Demoor

Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based
upload implementation.

The motivation is evidently performance: this would be beneficial for users with high network
bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store
(FYI: my contributions are made in name of Amplidata). 

This message was sent by Atlassian JIRA

View raw message