hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yonger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13637) improve setting of max connections in AWS client
Date Thu, 20 Jul 2017 12:11:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094592#comment-16094592

Yonger commented on HADOOP-13637:

[~steve_l] I don't think we should create a connection pool bigger than a work thread pool.
Assume the application run on Hadoop kick of request with X rate, each request cost T1 time
on thread and T1 time on connection, obviously, T1 will bigger than T2 due to a request always
get response to release a connection and then release a thread later . That means XT1>XT2,
and according little's law, N1(threads are needed concurrently) >N2 (connections are needed
concurrently). So the result is opposite of yours, we should set minimum thread pool size
base on max connection + n.

> improve setting of max connections in AWS client
> ------------------------------------------------
>                 Key: HADOOP-13637
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13637
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Priority: Minor
> things can go badly wrong if the S3A FS creates a thread pool for IO > than the number
of pooled AWS http connections (set by property MAXIMUM_CONNECTIONS); you also need some for
any other IO requests coming in.
> The max connections property is currently independent of thread pool size, and has a
default value of 1. 
> this is why there is a troubleshooting section in the docs showing the stack trace and
instructions to fix".
> Better: have a dynamic minimum like thread pool size + n, for a value of n to be chosen.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message