Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 23 Nov 2016 19:56:58 +0000 (UTC)
From: "Thomas Demoor (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13022295.1479759311000.354592.1479931018787@Atlassian.JIRA>
In-Reply-To: <JIRA.13022295.1479759311000@Atlassian.JIRA>
References: <JIRA.13022295.1479759311000@Atlassian.JIRA> <JIRA.13022295.1479759311861@arcas>
Subject: [jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy
 due to thread pool limits.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 23 Nov 2016 19:57:01 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15691219#comment-15691219 ] 

Thomas Demoor commented on HADOOP-13826:
----------------------------------------

Yes, that is set on the AmazonS3Client, which we use both for the TM as for our block uploads. The threadpools would share the connections, just as would let them share the number of active threads (as per my previous comment).

> S3A Deadlock in multipart copy due to thread pool limits.
> ---------------------------------------------------------
>
>                 Key: HADOOP-13826
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13826
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.7.3
>            Reporter: Sean Mackrory
>         Attachments: HADOOP-13826.001.patch, HADOOP-13826.002.patch
>
>
> In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The TransferManager javadocs (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html) explain how this is possible:
> {quote}It is not recommended to use a single threaded executor or a thread pool with a bounded work queue as control tasks may submit subtasks that can't complete until all sub tasks complete. Using an incorrectly configured thread pool may cause a deadlock (I.E. the work queue is filled with control tasks that can't finish until subtasks complete but subtasks can't execute because the queue is filled).{quote}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org