cxf-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Freeman Fang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CXF-7122) Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
Date Tue, 08 Nov 2016 06:42:02 GMT

    [ https://issues.apache.org/jira/browse/CXF-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15646690#comment-15646690
] 

Freeman Fang edited comment on CXF-7122 at 11/8/16 6:41 AM:
------------------------------------------------------------

Hi William,

In ahc 4.0.2 code base, the org.apache.http.impl.nio.conn.CPool is actually
{code}
class CPool extends AbstractNIOConnPool<HttpRoute, ManagedNHttpClientConnection, CPoolEntry>
{

    private final Log log = LogFactory.getLog(CPool.class);

    private final long timeToLive;
    private final TimeUnit tunit;

    public CPool(
            final ConnectingIOReactor ioreactor,
            final NIOConnFactory<HttpRoute, ManagedNHttpClientConnection> connFactory,
            final SocketAddressResolver<HttpRoute> addressResolver,
            final int defaultMaxPerRoute, final int maxTotal,
            final long timeToLive, final TimeUnit tunit) {
        super(ioreactor, connFactory, addressResolver, defaultMaxPerRoute, maxTotal);
        this.timeToLive = timeToLive;
        this.tunit = tunit;
    }

    @Override
    protected CPoolEntry createEntry(final HttpRoute route, final ManagedNHttpClientConnection
conn) {
        return new CPoolEntry(this.log, conn.getId(), route, conn, this.timeToLive, this.tunit);
    }

}
{code}
There's no onRelease method in that version, and I found HTTPASYNC-88(https://github.com/ok2c/httpasyncclient/commit/c246f59794cfaad86e718ad9bcc38cf368544d8a)
actually fixed this and this is in ahc 4.1.0 afterwards.

With ahc 4.1.0 afterwards, I think we don't need fix for CXF-6910 and CXF-7112(because ReceivedTimeout
we set on ahc RequestConfig won't affect the connection return to the pool), I will re-consider
your patch based on this.

Freeman




was (Author: ffang):
Hi William,

In ahc 4.0.2 code base, the org.apache.http.impl.nio.conn.CPool is actually
{code}
class CPool extends AbstractNIOConnPool<HttpRoute, ManagedNHttpClientConnection, CPoolEntry>
{

    private final Log log = LogFactory.getLog(CPool.class);

    private final long timeToLive;
    private final TimeUnit tunit;

    public CPool(
            final ConnectingIOReactor ioreactor,
            final NIOConnFactory<HttpRoute, ManagedNHttpClientConnection> connFactory,
            final SocketAddressResolver<HttpRoute> addressResolver,
            final int defaultMaxPerRoute, final int maxTotal,
            final long timeToLive, final TimeUnit tunit) {
        super(ioreactor, connFactory, addressResolver, defaultMaxPerRoute, maxTotal);
        this.timeToLive = timeToLive;
        this.tunit = tunit;
    }

    @Override
    protected CPoolEntry createEntry(final HttpRoute route, final ManagedNHttpClientConnection
conn) {
        return new CPoolEntry(this.log, conn.getId(), route, conn, this.timeToLive, this.tunit);
    }

}
{code}
There's no onRelease method in that version, and I found HTTPASYNC-88 actually fixed this
and this is in ahc 4.1.0 afterwards.

With ahc 4.1.0 afterwards, I think we don't need fix for CXF-6910 and CXF-7112(because ReceivedTimeout
we set on ahc RequestConfig won't affect the connection return to the pool), I will re-consider
your patch based on this.

Freeman



> Infinite loop due to AsyncHTTPConduit read timeout with exhausted connection pool
> ---------------------------------------------------------------------------------
>
>                 Key: CXF-7122
>                 URL: https://issues.apache.org/jira/browse/CXF-7122
>             Project: CXF
>          Issue Type: Bug
>          Components: Transports
>            Reporter: William Montaz
>            Assignee: Freeman Fang
>            Priority: Critical
>             Fix For: 3.2.0, 3.1.9
>
>         Attachments: AsyncHTTPConduitTest.java
>
>
> Using AsyncHTTPConduit, when the underlying connection pool gets exhausted, requests
waiting for a connection will lead to an infinite loop if they reach receive timeout.
> The problem occured on all versions of CXF above 3.0.5 (we did not tested other ones).

> Let's imagine a backend that's broken and leads to timeout for all requests.
> When handling requests, the cxf worker thread will eventually go in wait state (AsyncHTTPConduit:618),
with a timeout that matches the HTTPClientPolicy.setReceiveTimeout() value, waiting for the
NIO stack to complete and call notifyAll via responseCallback (AsyncHTTPConduit:455). 
> The timeout on the wait is the big problem :
> With our broken backend, the connection pool is exhausted waiting for other requests
to timeout. When a new request is made by cxf against this backend, after timeout time this
will happen :
>  - on the one side the reactor threads will get a connection from the pool and try to
write to the output stream. Waiting in the pool is not considered as receive timeout.
>  - on the other side the cxf worker thread will wake up (because of the timedout wait),
and shutdown SharedOutputBuffer and SharedInputBuffer (AsyncHTTPClient:624)
>  - reactor threads will go to infinite loop because they will try to produceContent from
a shutdown buffer (SharedOutputBuffer:120)
>  
>  From there, application recovery is compromised.
>   
>  To fix that, timeout should be handled only via the client callback (AsyncHTTPConduit:463).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message