cxf-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Freeman Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CXF-7122) Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
Date Mon, 07 Nov 2016 06:07:58 GMT

    [ https://issues.apache.org/jira/browse/CXF-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15643233#comment-15643233
] 

Freeman Fang commented on CXF-7122:
-----------------------------------

Hi William,

FYI, with your original patch, with the AsyncHTTPConduitTest I appended, you can run into
infinite loop also with 
AsyncHTTPConduitTest#testTimeout, so both the sync and async way has the infinite loop issue.
If you will re-work a solution, please also consider this.

Btw, the SocketTimeoutException name may not be appropriate for all case, we can reconsider
to use more accurate exception name like RecieveTimeoutException after the real issue get
resolved.

And if you wanna more help from community to resolve the issue you ran into, a reproducer
test case definitely would be more helpful.

Thanks 
Freeman

> Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
> ---------------------------------------------------------------------------------
>
>                 Key: CXF-7122
>                 URL: https://issues.apache.org/jira/browse/CXF-7122
>             Project: CXF
>          Issue Type: Bug
>          Components: Transports
>            Reporter: William Montaz
>            Assignee: Freeman Fang
>            Priority: Critical
>             Fix For: 3.2.0, 3.1.9
>
>         Attachments: AsyncHTTPConduitTest.java
>
>
> Using AsyncHTTPConduit, when the underlying connection pool gets exhausted, requests
waiting for a connection will lead to an infinite loop if they reach receive timeout.
> The problem occured on all versions of CXF above 3.0.5 (we did not tested other ones).

> Let's imagine a backend that's broken and leads to timeout for all requests.
> When handling requests, the cxf worker thread will eventually go in wait state (AsyncHTTPConduit:618),
with a timeout that matches the HTTPClientPolicy.setReceiveTimeout() value, waiting for the
NIO stack to complete and call notifyAll via responseCallback (AsyncHTTPConduit:455). 
> The timeout on the wait is the big problem :
> With our broken backend, the connection pool is exhausted waiting for other requests
to timeout. When a new request is made by cxf against this backend, after timeout time this
will happen :
>  - on the one side the reactor threads will get a connection from the pool and try to
write to the output stream. Waiting in the pool is not considered as receive timeout.
>  - on the other side the cxf worker thread will wake up (because of the timedout wait),
and shutdown SharedOutputBuffer and SharedInputBuffer (AsyncHTTPClient:624)
>  - reactor threads will go to infinite loop because they will try to produceContent from
a shutdown buffer (SharedOutputBuffer:120)
>  
>  From there, application recovery is compromised.
>   
>  To fix that, timeout should be handled only via the client callback (AsyncHTTPConduit:463).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message