Mailing-List: contact httpclient-users-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient User Discussion" <httpclient-users@hc.apache.org>
Message-ID: <1475695126.16158.1.camel@apache.org>
Subject: Re: AW: Why are many TCP open connections created on OS (Linux)?
From: Oleg Kalnichevski <olegk@apache.org>
To: HttpClient User Discussion <httpclient-users@hc.apache.org>
Cc: Bernd Eckenfels <ecki@zusammenkunft.net>
Date: Wed, 05 Oct 2016 21:18:46 +0200
In-Reply-To: <AC79553F4B34A546BB4C224AE0542B21245FE4D2@NTOVMAIL04.ad.otto.de>
References: <AC79553F4B34A546BB4C224AE0542B21245FE434@NTOVMAIL04.ad.otto.de>
	 <20161004211907.00003ef4.ecki@zusammenkunft.net>
	 <AC79553F4B34A546BB4C224AE0542B21245FE4D2@NTOVMAIL04.ad.otto.de>
Content-Type: text/plain; charset="UTF-8"
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit
archived-at: Wed, 05 Oct 2016 19:18:52 -0000

On Wed, 2016-10-05 at 12:16 +0000, Gnerlich, Manuel (IPS PROJECTS GMBH)
wrote:
> Hello Bernd,
> 
> thank you for your reply. 
> 
> I see them as ESTABLISHED and TIME_WAIT in netstat. The connections with ESTABLISHED state are ok and most under 100, but the connections with TIME_WAIT state are always more than 200. I mean on the client. On the server is ok and has little open connections. 

Please take a look at this resource:

http://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions

Oleg

> 
> Gruss ;) Manuel
> 
> Here some important  code, how I set it up:
> 
> public class NetworkClient implements AutoCloseable {
> ...
> public NetworkClient() {
>     ConnectingIOReactor ioReactor;
>     try {
>       // Create I/O reactor configuration
>       IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
>           .setIoThreadCount(Runtime.getRuntime().availableProcessors())
>           .setConnectTimeout(config.getConnectTimeout()).setSoTimeout(config.getSoTimeout())
>           .build();
>       ioReactor = new DefaultConnectingIOReactor(ioReactorConfig);
> 
>       PoolingNHttpClientConnectionManager cm = new PoolingNHttpClientConnectionManager(ioReactor);
>       cm.setMaxTotal(config.getConnectionMaxTotal());
>       cm.setDefaultMaxPerRoute(config.getConnectionDefaultMaxPerRoute());
> 
>       RequestConfig globalConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.IGNORE_COOKIES)
>           .build();
> 
>       HttpProcessor httpprocessor = HttpProcessorBuilder.create().build();
>       httpClient = HttpAsyncClients.custom().setDefaultRequestConfig(globalConfig)
>           .setHttpProcessor(httpprocessor).setConnectionManager(cm).build();
>     } catch (IOReactorException e) {
>       LOGGER.error("Could not create ConnectionIOReactor", e);
>     }
> 
>     initClientTransport();
>   }
> 
>   private void initClientTransport() {
>     httpClient.start();
>   }
> 
> ....
> ....
> public Integer send(String destination, int port, HttpRequest httpRequest, String operationName) {
>     Assert.notNull(destination);
>     Assert.notNull(httpRequest);
> 
>     startTime = startTime == 0 ? System.nanoTime() : startTime;
> 
>     HttpHost target = new HttpHost(destination, port);
> 
>     NcResponse ncResponse = getNewNcResponse(operationName);
> 
>     NcResponseProducer responseProducer = new NcResponseProducer(target, httpRequest, ncResponse,
>         requestCounter);
>     NcResponseConsumer responseConsumer = new NcResponseConsumer(ncResponse);
> 
>     httpClient.execute(responseProducer, responseConsumer, new ClientCallback(responseCounter,
>         failCounter, connectionErrorMessageList, getRequestLine(httpRequest)));
> 
>     long clientId = requestClientId.getAndIncrement();
>     return ((Long) clientId).intValue();
>   }
> ..
> }
> 
> public class NcResponseProducer extends BasicAsyncRequestProducer {
> 
>   private final NcResponse ncResponse;
> 
>   private static final Logger LOGGER = LogManager.getLogger();
> 
>   private final AtomicLong requestCounter;
> 
>   private final Header[] allHeaders;
> 
>   public NcResponseProducer(HttpHost target, HttpRequest request, NcResponse ncResponse,
>       AtomicLong requestCounter) {
>     super(target, request);
>     this.allHeaders = request.getAllHeaders();
>     this.ncResponse = ncResponse;
>     this.requestCounter = requestCounter;
>   }
> 
>   @Override
>   public void requestCompleted(HttpContext context) {
>     super.requestCompleted(context);
>     ncResponse.setStartTimeInNano(System.nanoTime());
>     LOGGER.debug("Sent request to client {}. request ID={} headers={} count={}",
>         getTarget().toHostString(), ncResponse.getRequestID(), Arrays.asList(allHeaders),
>         requestCounter.incrementAndGet());
>   }
> 
> }
> 
> public class NcResponseConsumer extends BasicAsyncResponseConsumer {
> 
>   private final NcResponse ncResponse;
> 
>   public NcResponseConsumer(NcResponse ncResponse) {
>     this.ncResponse = ncResponse;
>   }
> 
>   @Override
>   protected void onResponseReceived(HttpResponse response) throws IOException {
>     super.onResponseReceived(response);
>     ncResponse.setEndTimeInNano(System.nanoTime());
>     ncResponse.setReturnCode(response.getStatusLine().getStatusCode());
>   }
> 
> }
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Bernd Eckenfels [mailto:ecki@zusammenkunft.net] 
> Gesendet: Dienstag, 4. Oktober 2016 21:19
> An: Gnerlich, Manuel (IPS PROJECTS GMBH) <Manuel.Gnerlich@otto.de>
> Cc: httpclient-users@hc.apache.org
> Betreff: Re: Why are many TCP open connections created on OS (Linux)?
> 
> Am Tue, 4 Oct 2016 14:00:38 +0000
> schrieb "Gnerlich, Manuel (IPS PROJECTS GMBH)"
> <Manuel.Gnerlich@otto.de>:
> 
> > I am using Apache AsychHttpClient with connection pool to send more 
> > than 1 mio requests and configured maxinum number of connection (200) 
> > and of connection per Host (100).
> > 
> > However there are many open TCP connections (about 30.000) on linux.
> > Why? How can I control number of connection over Java on linux?
> 
> Define open for us, please. Do you mean you see them as ESTABLISHED in netstat or do you mean you see them as socket handles for the Java process (in lsof). Do you mean on the client or the server?
> 
> Can you show some code how you set it up and use the client.
> 
> Gruss
> Bernd
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org