Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE4A21789C for ; Fri, 27 Feb 2015 01:58:05 +0000 (UTC) Received: (qmail 62539 invoked by uid 500); 27 Feb 2015 01:58:05 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 62484 invoked by uid 500); 27 Feb 2015 01:58:05 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 62472 invoked by uid 99); 27 Feb 2015 01:58:05 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2015 01:58:05 +0000 Date: Fri, 27 Feb 2015 01:58:05 +0000 (UTC) From: "Steve Loughran (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-11183) Memory-based S3AOutputstream MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-11183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339584#comment-14339584 ] Steve Loughran commented on HADOOP-11183: ----------------------------------------- OK, doing a test run on this. Like you say, it's harmless unless enabled, and the docs now make that clear. The main risk is taking on the obligation to maintain it for an indefinite period. However, on the basis that we could always return the classic output stream, cut this code and simply print a warning if the flag is set, that's a low cost obligation. > Memory-based S3AOutputstream > ---------------------------- > > Key: HADOOP-11183 > URL: https://issues.apache.org/jira/browse/HADOOP-11183 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 2.6.0 > Reporter: Thomas Demoor > Assignee: Thomas Demoor > Attachments: HADOOP-11183-004.patch, HADOOP-11183-005.patch, HADOOP-11183-006.patch, HADOOP-11183.001.patch, HADOOP-11183.002.patch, HADOOP-11183.003.patch, design-comments.pdf > > > Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based upload implementation. > The motivation is evidently performance: this would be beneficial for users with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store (FYI: my contributions are made in name of Amplidata). -- This message was sent by Atlassian JIRA (v6.3.4#6332)