hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11183) Memory-based S3AOutputstream
Date Fri, 27 Feb 2015 01:58:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339584#comment-14339584

Steve Loughran commented on HADOOP-11183:

OK, doing a test run on this. Like you say, it's harmless unless enabled, and the docs now
make that clear.

The main risk is taking on the obligation to maintain it for an indefinite period. However,
on the basis that we could always return the classic output stream, cut this code and simply
print a warning if the flag is set, that's a low cost obligation. 

> Memory-based S3AOutputstream
> ----------------------------
>                 Key: HADOOP-11183
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11183
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Thomas Demoor
>            Assignee: Thomas Demoor
>         Attachments: HADOOP-11183-004.patch, HADOOP-11183-005.patch, HADOOP-11183-006.patch,
HADOOP-11183.001.patch, HADOOP-11183.002.patch, HADOOP-11183.003.patch, design-comments.pdf
> Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding
a memory-based upload implementation.
> The motivation is evidently performance: this would be beneficial for users with high
network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object
store (FYI: my contributions are made in name of Amplidata). 

This message was sent by Atlassian JIRA

View raw message