hc-httpclient-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Boniface <tho...@stickyads.tv>
Subject Re: Pool congestion
Date Mon, 18 May 2015 09:11:36 GMT
Thanks your answers, also hystrix seems pretty interesting. I'll have a
look into it.

Regarding my problem, I observed some cases where my application becomes
apparently stuck. After nothing happens in my application log for a couple
of seconds a thread dump is made. This thread dump showed that all I/O
dispatcher threads and all the http nio threads were waiting for the a lock
from AbstractNIOConnPool.

Here is an example:

"http-nio-127.0.0.1-8080-exec-100" daemon prio=10 tid=0x00007f7a64a21000
nid=0x56c2 waiting on condition [0x00007f7a4e8e6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ba27ded8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
        at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
        at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        at
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:271)
        at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:265)
        at
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:358)
        at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:125)
        at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
        at
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:74)
        at
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:107)
        at
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:91)
        at
com.stickyadstv.adex.bidder.openrtb.OpenRTBBuyerPlatform.sendBidRequest(OpenRTBBuyerPlatform.java:117)
        at
com.stickyadstv.adex.Auctioneer.sendBidRequests(Auctioneer.java:338)
        at com.stickyadstv.adex.Auctioneer.startAuction(Auctioneer.java:152)
        at
com.stickyadstv.adex.bidder.marketplace.MarketPlaceBuyerPlatform.startMarketPlaceAuction(MarketPlaceBuyerPlatform.java:144)
        at
com.stickyadstv.adex.bidder.marketplace.MarketPlaceBuyerPlatform.sendBidRequest(MarketPlaceBuyerPlatform.java:82)
        at
com.stickyadstv.adex.Auctioneer.sendBidRequests(Auctioneer.java:338)
        at com.stickyadstv.adex.Auctioneer.startAuction(Auctioneer.java:152)
        at
networkComm.commands.SwfIndexCommand.getProtocolSpecificResponse(SwfIndexCommand.java:66)
        at networkComm.commands.HttpCommand.getResponse(HttpCommand.java:70)
        at
com.stickyadstv.web.SwfIndexServlet.doGet(SwfIndexServlet.java:39)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
com.stickyadstv.deliveryengine.http.CORSAndNoCacheFilter.doFilter(CORSAndNoCacheFilter.java:34)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:620)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1700)
        - locked <0x00000000d4a6a4a0> (a
org.apache.tomcat.util.net.NioChannel)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)

Is this possible the lock leaked in a particular use case (this did not
happened at a request peak) ?

Thanks


2015-05-13 10:20 GMT+02:00 Oleg Kalnichevski <olegk@apache.org>:

> On Wed, 2015-05-13 at 09:16 +0200, Thomas Boniface wrote:
> > Thanks for the pointers, this does seems similar to what I was thinking
> of.
> > My understanding is that this is currently not possible with the http
> async
> > client at the moment. Applying this logic to the httpasyncclient would
> mean
> > dynamically change the route max connection depending on how well they
> > behave. Is that right ?
> >
>
> Sounds correct.
>
> Oleg
>
> > Thomas
> >
> > 2015-05-08 9:46 GMT+02:00 Oleg Kalnichevski <olegk@apache.org>:
> >
> > > On Thu, 2015-05-07 at 11:22 +0200, Thomas Boniface wrote:
> > > > Hi,
> > > >
> > > > I have an application that receives and HTTP request from a user and
> > > > contact multiple external servers. When each external server
> responded
> > > (or
> > > > the servlet timeout is reached) and HTTP response is built based on
> > > > external servers responses.
> > > >
> > > > I think in such a case, when one of the external server is having
> trouble
> > > > to respond as fast as it should my application will become less and
> less
> > > > responsive: as it will wait to reach servlet timeout to respond to
> the
> > > > client and other client request still incoming will experience the
> same
> > > > problem.
> > > >
> > > > Thinking on how to prevent such cases, I started thinking of ways to
> > > > decrease the number of requests made to route having bad
> performances (by
> > > > implementing an exponential backoff mechanism for instance) but it
> came
> > > to
> > > > my mind that it may be possible to prevent this just by modifying the
> > > pool
> > > > configuration. My idea would be to greatly decrease the connection
> > > request
> > > > timeout (setConnectionRequestTimeout), my understanding is if the
> http
> > > > async client cannot send the request within say 5ms it probably
> means the
> > > > route is currently overloaded.
> > > >
> > > > Is this the right approach to solve this type of scenario ?
> > > >
> > > > Thanks,
> > > > Thomas
> > >
> > > Thomas
> > >
> > > HttpClient ships with a so called AIMD back-off manager
> > >
> > >
> http://hc.apache.org/httpcomponents-client-4.4.x/httpclient/apidocs/org/apache/http/impl/client/AIMDBackoffManager.html
> > >
> > >
> http://hc.apache.org/httpcomponents-client-4.4.x/httpclient/apidocs/org/apache/http/impl/client/DefaultBackoffStrategy.html
> > >
> > > You probably might be able to use it (or some custom implementation)
> for
> > > your purpose.
> > >
> > > Oleg
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message