hc-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Oleg Kalnichevski (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HTTPCLIENT-1809) Thread contention in PoolingHttpClientConnectionManager
Date Sat, 28 Jan 2017 10:42:24 GMT

    [ https://issues.apache.org/jira/browse/HTTPCLIENT-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15844013#comment-15844013
] 

Oleg Kalnichevski edited comment on HTTPCLIENT-1809 at 1/28/17 10:41 AM:
-------------------------------------------------------------------------

Andrew,
Pool lock contention is a known issue. We do get similar reports every one in a while. However
in most cases that I had a chance to take a look at it turned out to be a configuration issue
or a design flaw: having too many threads contend for a few connections or having thousands
of threads whereas a hundred threads would be sufficient to process the same amount of requests
per second. Having said that we do have plans to build a lock-less pool implementation at
some point (see HTTPCORE-390) but right now I have no bandwidth to do it myself. I am in the
process of refactoring the connection management code in trunk (5.0.x branch) and making sure
that that all potentially expensive operations are executed outside the pool lock. This should
help reduce lock contention. 

If I were to look into this problem I would start off by finding out whether the application
actually needs so many threads to begin with and only then proceed with an idea of building
a custom connection manager.

Oleg  


was (Author: olegk):
Andrew,
Pool lock contention is a known issue. We do get similar reports every one in a while. However
in most cases that I had a chance to take a look at it turned out to be a configuration issue
or a design flaw: having too many threads contend for a few connections or having thousands
of threads whereas a hundred threads would be sufficient to process the same amount of requests
per second. Having said that we do have plans to build a lock-less pool implementation at
some point (see HTTPCORE-390) but right now I have no bandwidth to do it myself. I am in the
process of refactoring the connection management code in trunk (5.0.x branch) and making sure
that that all potentially expensive operations are executed out of the pool lock. This should
help reduce lock contention. 

If I were to look into this problem I would start off by finding out whether the application
actually needs so many threads to begin with and only then proceed with an idea of building
a custom connection manager.

Oleg  

> Thread contention in PoolingHttpClientConnectionManager
> -------------------------------------------------------
>
>                 Key: HTTPCLIENT-1809
>                 URL: https://issues.apache.org/jira/browse/HTTPCLIENT-1809
>             Project: HttpComponents HttpClient
>          Issue Type: Bug
>          Components: HttpClient (classic)
>    Affects Versions: 4.5
>            Reporter: Andrew Shore
>            Priority: Minor
>              Labels: perfomance
>
> We (AWS SDK for Java) have been investigating reports of poor performance in the SDK
and have narrowed it down to thread contention issues in PoolingHttpClientConnectionManager.
Up to a certain TPS, performance is great and their is no issue. After a certain TPS (approx
8000 in our load tests), performance tanks hard and most threads end up stuck waiting on a
lock in AbstractConnPool (in either lease or releaseConnection).
> https://github.com/apache/httpcore/blob/4.4.x/httpcore/src/main/java/org/apache/http/pool/AbstractConnPool.java#L403
> This quickly locks up the application as it tries to meet the incoming TPS. We have been
able to workaround this and achieve much higher throughput but having multiple SDK clients
and round robin selecting them to hand off to threads. This allowed us to easily scale up
to 16, 000 TPS. We wanted to open up a dialog with the maintainers of the Apache HTTP client
to see if this is a known issue/limitation and what options we have for getting around it.
We aren’t opposed to re-implementing the connection manager to be more performant but since
it’s a pretty sizable chunk of work we wanted to ensure that’s the best path forward before
proceeding. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org


Mime
View raw message