hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11684) S3a to use thread pool that blocks clients
Date Thu, 24 Sep 2015 19:00:06 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906832#comment-14906832
] 

Steve Loughran commented on HADOOP-11684:
-----------------------------------------

bq. Anyone have example where many threads would be entering S3AFilesystem to do writes

any multi-threaded process with lots of workers committing the output of their work to s3.
Examples: MR, Tez, Spark work

HADOOP-11446 caught this with HBase FS snapshots.

bq.  Since the behavior of these parameters has changed, should we add an entry to release
notes notifying folks of the new behavior?

s3a has still been in stabilisation phase, so I'm not so sure as to how much we need to detail.
the main thing is "do the changes consistute some form of regression?". If they improve things,
then they aren't. After all, currently when the thread pool fills up, you are in trouble —so
you do need to overallocate

> S3a to use thread pool that blocks clients
> ------------------------------------------
>
>                 Key: HADOOP-11684
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11684
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.7.0
>            Reporter: Thomas Demoor
>            Assignee: Thomas Demoor
>         Attachments: HADOOP-11684-001.patch, HADOOP-11684-002.patch, HADOOP-11684-003.patch
>
>
> Currently, if fs.s3a.max.total.tasks are queued and another (part)upload wants to start,
a RejectedExecutionException is thrown. 
> We should use a threadpool that blocks clients, nicely throtthling them, rather than
throwing an exception. F.i. something similar to https://github.com/apache/incubator-s4/blob/master/subprojects/s4-comm/src/main/java/org/apache/s4/comm/staging/BlockingThreadPoolExecutorService.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message