hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Genmao Yu (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
Date Tue, 14 Nov 2017 05:38:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Genmao Yu updated HADOOP-14999:
-------------------------------
    Comment: was deleted

(was: [~stevel@apache.org] and [~drankye] Any feedback is appreciated)

> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-14999
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14999
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/oss
>    Affects Versions: 3.0.0-beta1
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>         Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch
>
>
> This mechanism is designed for uploading file in parallel and asynchronously: 
> - improve the performance of uploading file to OSS server. Firstly, this mechanism splits
result to multiple small blocks and upload them in parallel. Then, getting result and uploading
blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme example, there
is a task which will output 100GB or even larger, we may need to output this 100GB to local
disk and then upload it. Sometimes, it is inefficient and limited to disk space.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message