hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11570) S3AInputStream.close() downloads the remaining bytes of the object from S3
Date Tue, 17 Feb 2015 16:43:14 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324423#comment-14324423
] 

Hudson commented on HADOOP-11570:
---------------------------------

FAILURE: Integrated in Hadoop-trunk-Commit #7128 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7128/])
HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3.
(Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)
* hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
* hadoop-common-project/hadoop-common/CHANGES.txt


> S3AInputStream.close() downloads the remaining bytes of the object from S3
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-11570
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11570
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Dan Hecht
>            Assignee: Dan Hecht
>             Fix For: 2.7.0
>
>         Attachments: HADOOP-11570-001.patch, HADOOP-11570-002.patch
>
>
> Currently, S3AInputStream.close() calls S3Object.close().  But, S3Object.close() will
read the remaining bytes of the S3 object, potentially transferring a lot of bytes from S3
that are discarded.  Instead, the wrapped stream should be aborted to avoid transferring discarded
bytes (unless the preceding read() finished at contentLength).  For example, reading only
the first byte of a 1 GB object and then closing the stream will result in all 1 GB transferred
from S3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message