hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients
Date Thu, 01 Oct 2015 01:22:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14939139#comment-14939139
] 

Aaron Fabbri commented on HADOOP-11684:
---------------------------------------

Ok.  With that explanation I'm fine with the new defaults.  Worst case a config change can
be made to re-tune.

I'm a little confused about the JVM out of memory (OOM) issue: Either way, you have many threads
in play.  Whether they block, or start uploading themselves. Is there something about uploading
that dramatically increases the memory usage per thread?

Certainly, the blocking executor is more robust at limiting the number of concurrent uploads.
 I like it, it just takes more effort for me to understand it is bug-free.

> S3a to use thread pool that blocks clients
> ------------------------------------------
>
>                 Key: HADOOP-11684
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11684
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.7.0
>            Reporter: Thomas Demoor
>            Assignee: Thomas Demoor
>         Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch, HADOOP-11684-003.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload wants to start,
a RejectedExecutionException is thrown. 
> We should use a threadpool that blocks clients, nicely throtthling them, rather than
throwing an exception. F.i. something similar to https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message