hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Demoor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.
Date Wed, 23 Nov 2016 08:45:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15689382#comment-15689382
] 

Thomas Demoor commented on HADOOP-13826:
----------------------------------------

Mmmh, sorry,  I caused this when introducing the bounded pool to protect against OOM when
streaming the upload from a memory buffer. To my defense, at time of implementation we were
programming against aws-sdk 1.7.4 where the docs don't seem to mention this :( 

The bounded threadpool blocks the clients (data producers) so memory usage is bounded. The
TransferManager has built in throttling so they do not need this and advocate using an unbounded
pool. However, the TransferManager does not offer functionality to do "block-based" uploads.
So for S3ABlockOutputStream we need a different solution. The first thing that came to my
mind was two threadpools (1 unbounded for TransferManager, 1 bounded for Block) which have
a shared limit for number of active (uploading) threads but would like something simpler if
possible. Have not yet studied [~mackrorysd]'s solution in detail, will have a look.

> S3A Deadlock in multipart copy due to thread pool limits.
> ---------------------------------------------------------
>
>                 Key: HADOOP-13826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.3
>            Reporter: Sean Mackrory
>         Attachments: HADOOP-13826.001.patch, HADOOP-13826.002.patch
>
>
> In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The TransferManager
javadocs (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html)
explain how this is possible:
> {quote}It is not recommended to use a single threaded executor or a thread pool with
a bounded work queue as control tasks may submit subtasks that can't complete until all sub
tasks complete. Using an incorrectly configured thread pool may cause a deadlock (I.E. the
work queue is filled with control tasks that can't finish until subtasks complete but subtasks
can't execute because the queue is filled).{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message