hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14081) S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
Date Mon, 20 Feb 2017 15:06:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15874662#comment-15874662
] 

Steve Loughran commented on HADOOP-14081:
-----------------------------------------

Patch LGTM except for where there's a new test which does a 6MB upload. We dont need this
as the huge block scale tests do large uploads. If we did need it, then it MUST go into a
scale {{ITestHuge*}} test, and so run serialized explicitly when scale is enabled. And take
configurable upload sizes for better testing. But as it doesn't add anything to our huge tests,
it's not needed here

Did you run the scale tests with this patch? I'm going to apply the patch, revert the test
patch and do that test run now. 

> S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock)
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-14081
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14081
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HADOOP-14081.001.patch
>
>
> In {{S3ADataBlocks::ByteArrayBlock}}, data is copied whenever {{startUpload}} is called.
It might be possible to directly access the byte[] array from ByteArrayOutputStream. 
> Might have to extend ByteArrayOutputStream and create a method like getInputStream()
which can return ByteArrayInputStream.  This would avoid expensive array copy during large
upload.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message