hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-17139) Re-enable optimized copyFromLocal implementation in S3AFileSystem
Date Mon, 02 Aug 2021 15:13:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran resolved HADOOP-17139.
    Fix Version/s: 3.3.2
       Resolution: Fixed

added in 3.3.2 Hadoop FS Shell uploads to S3 just got a lot faster!

> Re-enable optimized copyFromLocal implementation in S3AFileSystem
> -----------------------------------------------------------------
>                 Key: HADOOP-17139
>                 URL: https://issues.apache.org/jira/browse/HADOOP-17139
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0, 3.2.1
>            Reporter: Sahil Takiar
>            Assignee: Bogdan Stolojan
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 3.3.2
>          Time Spent: 9h 50m
>  Remaining Estimate: 0h
> It looks like HADOOP-15932 disabled the optimized copyFromLocal implementation in S3A
for correctness reasons.  innerCopyFromLocalFile should be fixed and re-enabled. The current
implementation uses FileSystem.copyFromLocal which will open an input stream from the local
fs and an output stream to the destination fs, and then call IOUtils.copyBytes. With default
configs, this will cause S3A to read the file into memory, write it back to a file on the
local fs, and then when the file is closed, upload it to S3.
> The optimized version of copyFromLocal in innerCopyFromLocalFile, directly creates a
PutObjectRequest request with the local file as the input.

This message was sent by Atlassian Jira

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message