Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Message-ID: <1465573702.31151.9.camel@apache.org>
Subject: Re: ManagedNHttpClientConnectionImpl Shutdown Vs. Close
From: Oleg Kalnichevski <olegk@apache.org>
To: "httpclient-users@hc.apache.org" <httpclient-users@hc.apache.org>
Date: Fri, 10 Jun 2016 17:48:22 +0200
In-Reply-To: <FEC30B23-0841-4BD3-B229-0DDE0E8BF849@mathworks.com>
References: <FEC30B23-0841-4BD3-B229-0DDE0E8BF849@mathworks.com>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
archived-at: Fri, 10 Jun 2016 15:48:29 -0000

On Fri, 2016-06-10 at 14:15 +0000, Poonam Chawla wrote:
> Hello Oleg, 
> 
> 
> I am using Apache HttpAsycnClient 4.1 and I noticed that the
> connections are not released on calling close.
> Here is what I am trying to do: (Please note that it is not a
> compileable code as I had to remove lines that are internal to my
> company). 
> 
> 
> - Create CloseableHttpAsyncClient
> 
> 
> CloseableHttpAsyncClient httpClientAsync;
> 
> 
> SSLIOSessionStrategy sslsf = new SSLIOSessionStrategy(sslContext,
>                     hostnameVerifier) {
>                 @Override
>                 protected void verifySession(HttpHost host,
>                         org.apache.http.nio.reactor.IOSession
> iosession,
>                         SSLSession sslsession) throws SSLException {
>                     verifySSLSession(iosession, sslsession,
> sslConfig);
>                 }
>             };
> 
> 
>             Registry<SchemeIOSessionStrategy> sessionStrategyRegistry
> = RegistryBuilder
>                     .<SchemeIOSessionStrategy> create()
>                     .register("http", NoopIOSessionStrategy.INSTANCE)
>                     .register("https", sslsf).build();
> 
> 
>             IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
>                     .setConnectTimeout((int)
> clientConfig.getTimeOutMs())
>                     .setSoTimeout((int) clientConfig.getTimeOutMs())
>                     .setTcpNoDelay(false).build();
> 
> 
>             ConnectingIOReactor ioReactor = new
> DefaultConnectingIOReactor(
>                     ioReactorConfig);
> 
> 
>             NHttpConnectionFactory<ManagedNHttpClientConnection>
> connFactory = new ManagedNHttpClientConnectionFactory();
>             PoolingNHttpClientConnectionManager connManager = new
> PoolingNHttpClientConnectionManager(
>                     ioReactor,connFactory, sessionStrategyRegistry);
>             connManager.setDefaultMaxPerRoute(clientConfig
>                     .getMaxConnectionsPerAddress());
> 
> 
>             httpClientAsync = HttpAsyncClients.custom()
>                     .setUserAgent(ApacheHttpClient.class.getName())
>                     .setConnectionManager(connManager)
>                     .setSSLContext(sslContext)
>                     .setSSLHostnameVerifier(hostnameVerifier)
>                     .setSSLStrategy(sslsf).build();
>             httpClientAsync.start();
> 
> 
> 
> 
> - Execute HTTPPosts using the httpClientAsync 
> 
> 
> try{ 
> // underneath creates the CloseableHttpAsyncClient
> MWHttpClient client = new MWHttpClient();
> 
> //underneath calls httpClientAsync.execute(….) 
> Future<> f = client.invoke(….)    
> f.get()
> 
> 
> //underneath calls httpClientAsync.execute(….)
> f1 = httpClientAsync.execute(…)
> 
> 
> catch{
> }finally{
> // spawn a new thread to close the httpClientAsync
> // If there are pending requests waiting for a response and the user
> calls close,
> // abort the pending requests using request.abort();
> // underneath calls httpClientAsync.close();
> 
> 
> client.close();
> }
> 
> 
> 
> 
> Here is a part of the entire log I am confused about:
> 
> 
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> GET /~81b806db-4546-4a7d-b623-6f0848ff6b05/requests?since=2448&format=protobuf&clients=97628818-0daf-405c-a14c-4891738587b7 HTTP/1.1
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Host:
> localhost:53097
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> Connection: Keep-Alive
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> User-Agent: com.mathworks.mps.client.internal.ApacheHttpClient
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: Event
> set [w]
> 2016/06/09 16:30:34:753 EDT [DEBUG] MainClientExec - [exchange: 12]
> Request completed
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: 247
> bytes written
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Event
> cleared [w]
> 2016/06/09 16:30:34:852 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] Cancelled
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]:
> Shutdown
> 2016/06/09 16:30:34:853 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager is shutting
> down
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
> 2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] connection aborted
> 2016/06/09 16:30:34:854 EDT [DEBUG] InternalIODispatch -
> http-outgoing-0 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG] InternalIODispatch -
> http-outgoing-1 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager shut down
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Releasing connection: [id:
> http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
> 0; route allocated: 0 of 10; total allocated: 0 of 20]
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection released: [id:
> http-outgoing-0][route: {}->http://localhost:53097][total kept alive:
> 0; route allocated: 0 of 10; total allocated: 0 of 20]
>  
> 
> Even though the log at the end states that there are 0 of 20
> connections allocated, this is what I see in the “Open files and
> ports” on Mac:
> ./.mps_socket/mathworks-mds-local-socket.83924.1
> ->0x60dc6d646a6b4acd
> localhost:53097->localhost:54726
> 

Please note that the OS may still keep the underlying TCP connection
open in TIME-WAIT even if the JVM considers the socket of that
connection fully closed.

> 
> I tried to run this code in a loop and what is interesting is any
> connection that is “SHUTDOWN” is still open when you look at the
> activity monitor, whereas any connection that is “CLOSED” is no longer
> open.
> 

I am not sure I understand this bit. Could you please provide a test app
demonstrating the issue?

Oleg 

> 
> This leak is eventually causing failures on our side with “Too many
> files open” error. 
> I would really appreciate if you could help me find a workaround to
> this issue.
> 
> 
> Thanks,
> Poonam
> 
> 
> 
> 
> 
> 
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org