hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Genmao Yu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-14999) AliyunOSS: provide one asynchronous multi-part based uploading mechanism
Date Tue, 14 Nov 2017 11:10:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250906#comment-16250906
] 

Genmao Yu edited comment on HADOOP-14999 at 11/14/17 11:09 AM:
---------------------------------------------------------------

pending on refactoring: use {{SemaphoredDelegatingExecutor}} instead of {{TaskEngine}}.

Just as discussed in HADOOP-15027, I think {{org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor}}
is a good common class, and we may move it to hadoop-common. [~stevel@apache.org] Do you mind
if I open jira to do this work?


was (Author: unclegen):
pending on refactoring: move the {{TaskEngine}} from output stream to oss filesystem .

Just as discussed in HADOOP-15027, I think {{org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor}}
is a good common class, and we may move it to hadoop-common. 
Then, I will refactor the {{TaskEngine}} to use {{SemaphoredDelegatingExecutor}}.

[~stevel@apache.org] Do you mind if I open jira to do this work?

> AliyunOSS: provide one asynchronous multi-part based uploading mechanism
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-14999
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14999
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/oss
>    Affects Versions: 3.0.0-beta1
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>         Attachments: HADOOP-14999.001.patch, HADOOP-14999.002.patch
>
>
> This mechanism is designed for uploading file in parallel and asynchronously: 
> - improve the performance of uploading file to OSS server. Firstly, this mechanism splits
result to multiple small blocks and upload them in parallel. Then, getting result and uploading
blocks are asynchronous.
> - avoid buffering too large result into local disk. To cite an extreme example, there
is a task which will output 100GB or even larger, we may need to output this 100GB to local
disk and then upload it. Sometimes, it is inefficient and limited to disk space.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message