cxf-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "William Montaz (JIRA)" <>
Subject [jira] [Commented] (CXF-7122) Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
Date Mon, 07 Nov 2016 10:20:59 GMT


William Montaz commented on CXF-7122:

Hi Freeman, Sergey,

The test you provide is a good one. But running it I saw something weird -> even when timeouting,
the requests where made against the server and that led me to the reason why the patch I propose
was not working on or 3.2-SNAPSHOT

Actually, I first tested my patch on version 3.0.5 of CXF and when backporting to 3.1.x I
did not realised a little thing changed in the AsyncHTTPConduit. As you can notice, on version
line 220, the RequestConfig sets a SocketTimeout equals to ReceiveTimeout.

Since my patch relies on using the callback to timeout, it could not work because the underlying
socket was not timed out. I think it is also the reason why ReadTimeout was not used when

I updated my merge request to add SocketTimeout.  I also notice there is a ConnectionRequestTimeout,
that helps dealing whit waiting for a connection for too long when the pool is exhausted.
In my merge request I propose to set it to ReceiveTimeout too, to stay close to the behavior
proposed by CXF. But we could actually set a new property to distinguish both.

I remove the timer task from Freeman's commit, just as a comment. You want also want to clean
this code in a better way.

Last thing, my project uses version 3.0.x, could we consider backporting this patch on 3.0.x


> Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
> ---------------------------------------------------------------------------------
>                 Key: CXF-7122
>                 URL:
>             Project: CXF
>          Issue Type: Bug
>          Components: Transports
>            Reporter: William Montaz
>            Assignee: Freeman Fang
>            Priority: Critical
>             Fix For: 3.2.0, 3.1.9
>         Attachments:
> Using AsyncHTTPConduit, when the underlying connection pool gets exhausted, requests
waiting for a connection will lead to an infinite loop if they reach receive timeout.
> The problem occured on all versions of CXF above 3.0.5 (we did not tested other ones).

> Let's imagine a backend that's broken and leads to timeout for all requests.
> When handling requests, the cxf worker thread will eventually go in wait state (AsyncHTTPConduit:618),
with a timeout that matches the HTTPClientPolicy.setReceiveTimeout() value, waiting for the
NIO stack to complete and call notifyAll via responseCallback (AsyncHTTPConduit:455). 
> The timeout on the wait is the big problem :
> With our broken backend, the connection pool is exhausted waiting for other requests
to timeout. When a new request is made by cxf against this backend, after timeout time this
will happen :
>  - on the one side the reactor threads will get a connection from the pool and try to
write to the output stream. Waiting in the pool is not considered as receive timeout.
>  - on the other side the cxf worker thread will wake up (because of the timedout wait),
and shutdown SharedOutputBuffer and SharedInputBuffer (AsyncHTTPClient:624)
>  - reactor threads will go to infinite loop because they will try to produceContent from
a shutdown buffer (SharedOutputBuffer:120)
>  From there, application recovery is compromised.
>  To fix that, timeout should be handled only via the client callback (AsyncHTTPConduit:463).

This message was sent by Atlassian JIRA

View raw message