hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hecht (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11570) S3AInputStream.close() downloads the remaining bytes of the object from S3
Date Thu, 12 Feb 2015 18:37:12 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318722#comment-14318722
] 

Dan Hecht commented on HADOOP-11570:
------------------------------------

Hi Steve, I've added patch 002 that does the threshold optimization for close. I'm okay with
this, but don't know whether this is important in practice or how much it helps.  It'll be
pretty workload dependent, but it is simple enough so maybe worthwhile.  Which patch do you
prefer?

In any case, I reran the S3A tests using patch 002 as well:
{code}
------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 233.817 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.897 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.383 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.457 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 5.925 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.815 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.911 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.859 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.593 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.036 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir

Results :

Tests run: 96, Failures: 0, Errors: 0, Skipped: 3
{code}

> S3AInputStream.close() downloads the remaining bytes of the object from S3
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-11570
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11570
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Dan Hecht
>            Assignee: Dan Hecht
>         Attachments: HADOOP-11570-001.patch, HADOOP-11570-002.patch
>
>
> Currently, S3AInputStream.close() calls S3Object.close().  But, S3Object.close() will
read the remaining bytes of the S3 object, potentially transferring a lot of bytes from S3
that are discarded.  Instead, the wrapped stream should be aborted to avoid transferring discarded
bytes (unless the preceding read() finished at contentLength).  For example, reading only
the first byte of a 1 GB object and then closing the stream will result in all 1 GB transferred
from S3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message