Mailing-List: contact dev-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpComponents Project" <dev@hc.apache.org>
Date: Sun, 26 Mar 2017 20:44:42 +0000 (UTC)
From: "Sekhar M (JIRA)" <jira@apache.org>
To: dev@hc.apache.org
Message-ID: <JIRA.13042884.1487066027000.126418.1490561082097@Atlassian.JIRA>
In-Reply-To: <JIRA.13042884.1487066027000@Atlassian.JIRA>
References: <JIRA.13042884.1487066027000@Atlassian.JIRA> <JIRA.13042884.1487066027777@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HTTPCORE-446) Deadlock in AbstractConnPool on
 shutdown
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 26 Mar 2017 20:44:52 -0000


    [ https://issues.apache.org/jira/browse/HTTPCORE-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942443#comment-15942443 ] 

Sekhar M commented on HTTPCORE-446:
-----------------------------------

I'm having the same issue. I'm using Apache Http Client for making http calls to a third party system and during load several requests are failing with this error. PoolingHttpClientConnectionManager is throwing the following exception while leasing connection from pool. Using http-core-4.4.4 and http-client-4.5.1 versions.
Can you help confirm if the following error is resolved in http-core 4.4.7?. Thanks in advance for any help.

{code}
org.apache.http.impl.execchain.RequestAbortedException: Request aborted
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:193) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) ~[httpclient-4.5.1.jar:4.5.1]
	at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:294) [hystrix-core-1.4.26.jar:1.4.26]
	at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:289) [hystrix-core-1.4.26.jar:1.4.26]
	at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]
	at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]
	at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]
	at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]
	at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]
	at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]
	at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]
	at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.4.26.jar:1.4.26]
	at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:499) [hystrix-core-1.4.26.jar:1.4.26]
	at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]
	at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1]
	at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.4.26.jar:1.4.26]
	at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.4.26.jar:1.4.26]
	at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.4.26.jar:1.4.26]
	at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_51]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
	at java.lang.Thread.null(Unknown Source) [na:1.8.0_51]
Caused by: java.lang.InterruptedException: null
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[na:1.8.0_51]
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[na:1.8.0_51]
	at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138) ~[httpcore-4.4.4.jar:4.4.4]
	at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306) ~[httpcore-4.4.4.jar:4.4.4]
	at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64) ~[httpcore-4.4.4.jar:4.4.4]
	at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192) ~[httpcore-4.4.4.jar:4.4.4]
	at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185) ~[httpcore-4.4.4.jar:4.4.4]
	at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107) ~[httpcore-4.4.4.jar:4.4.4]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263) ~[httpclient-4.5.1.jar:4.5.1]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[httpclient-4.5.1.jar:4.5.1]
	... 32 common frames omitted
{code}

> Deadlock in AbstractConnPool on shutdown
> ----------------------------------------
>
>                 Key: HTTPCORE-446
>                 URL: https://issues.apache.org/jira/browse/HTTPCORE-446
>             Project: HttpComponents HttpCore
>          Issue Type: Bug
>          Components: HttpCore
>    Affects Versions: 4.4.6
>            Reporter: Kiran G
>            Assignee: Oleg Kalnichevski
>            Priority: Minor
>             Fix For: 4.4.7
>
>
> Application on which the deadlock is seen is using PoolingHttpClientConnectionManager for managing http connections for HttpClient. There is a timer job which keeps executing HTTP requests on external urls, which is using this PoolingHttpClientConnectionManager.
> Sometimes while shutting down this application a deadlock is seen. Stack trace below.
> h4. Thread 1
> h5. Monitor 
> Owns Monitor Lock on com/ibm/ws/runtime/component/ApplicationMgrImpl@0x0000000700B17A58 
> Waiting for Monitor Lock on org/apache/http/pool/AbstractConnPool$2@0x0000000709E27898
> h5. Java Stack
> {code}
> at org/apache/http/pool/AbstractConnPool$2.cancel(AbstractConnPool.java:206) 
> at org/apache/http/pool/RouteSpecificPool.shutdown(RouteSpecificPool.java:155) 
> at org/apache/http/pool/AbstractConnPool.shutdown(AbstractConnPool.java:152) 
> at org/apache/http/impl/conn/PoolingHttpClientConnectionManager.shutdown(PoolingHttpClientConnectionManager.java:396) 
> at org/apache/http/impl/client/HttpClientBuilder$2.close(HttpClientBuilder.java:1225) 
> at org/apache/http/impl/client/InternalHttpClient.close(InternalHttpClient.java:201) 
> at com/test/monitor/HttpClientFactory.safeClose(HttpClientFactory.java:89) 
> at com/test/monitor/HttpClientFactory.close(HttpClientFactory.java:83) 
> at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) 
> at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76(Compiled Code)) 
> at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43(Compiled Code)) 
> at java/lang/reflect/Method.invoke(Method.java:607(Compiled Code)) 
> at org/springframework/beans/factory/annotation/InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) 
> at org/springframework/beans/factory/annotation/InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeDestroyMethods(InitDestroyAnnotationBeanPostProcessor.java:325) 
> at org/springframework/beans/factory/annotation/InitDestroyAnnotationBeanPostProcessor.postProcessBeforeDestruction(InitDestroyAnnotationBeanPostProcessor.java:154) 
> at org/springframework/beans/factory/support/DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:253) 
> at org/springframework/beans/factory/support/DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578) 
> at org/springframework/beans/factory/support/DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554) 
> at org/springframework/beans/factory/support/DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961) 
> at org/springframework/beans/factory/support/DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523) 
> at org/springframework/beans/factory/support/DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968) 
> at org/springframework/context/support/AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1033) 
> at org/springframework/context/support/AbstractApplicationContext.doClose(AbstractApplicationContext.java:1009) 
> at org/springframework/context/support/AbstractApplicationContext.close(AbstractApplicationContext.java:961) 
> at org/springframework/web/context/ContextLoader.closeWebApplicationContext(ContextLoader.java:581) 
> at org/springframework/web/context/ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116) 
> at com/ibm/ws/webcontainer/webapp/WebApp.notifyServletContextDestroyed(WebApp.java:1741) 
> at com/ibm/ws/webcontainer/webapp/WebApp.destroy(WebApp.java:3085) 
> at com/ibm/ws/webcontainer/webapp/WebAppImpl.destroy(WebAppImpl.java:1291) 
> at com/ibm/ws/container/AbstractContainer.destroy(AbstractContainer.java:75) 
> at com/ibm/ws/webcontainer/webapp/WebGroup.destroy(WebGroup.java:228) 
> at com/ibm/ws/webcontainer/webapp/WebGroup.removeWebApplication(WebGroup.java:269) 
> at com/ibm/ws/webcontainer/VirtualHost.removeWebApplication(VirtualHost.java:296) 
> at com/ibm/ws/webcontainer/VirtualHostImpl.removeWebApplication(VirtualHostImpl.java:211) 
> at com/ibm/ws/webcontainer/WSWebContainer.removeWebApplication(WSWebContainer.java:820) 
> at com/ibm/ws/webcontainer/component/WebContainerImpl.uninstall(WebContainerImpl.java:458) 
> at com/ibm/ws/webcontainer/component/WebContainerImpl.stop(WebContainerImpl.java:729) 
> at com/ibm/ws/runtime/component/ApplicationMgrImpl.stop(ApplicationMgrImpl.java:1216) 
> at com/ibm/ws/runtime/component/DeployedApplicationImpl.fireDeployedObjectStop(DeployedApplicationImpl.java:1375) 
> at com/ibm/ws/runtime/component/DeployedModuleImpl.stop(DeployedModuleImpl.java:671) 
> at com/ibm/ws/runtime/component/DeployedApplicationImpl.stop(DeployedApplicationImpl.java:1149) 
> at com/ibm/ws/runtime/component/ApplicationMgrImpl.stopApplication(ApplicationMgrImpl.java:950) 
> at com/ibm/ws/runtime/component/ApplicationMgrImpl.stopApplicationImpl(ApplicationMgrImpl.java:1479) 
> at com/ibm/ws/runtime/component/ApplicationMgrImpl.stop(ApplicationMgrImpl.java:2221) 
> at com/ibm/ws/runtime/component/CompositionUnitMgrImpl.stop(CompositionUnitMgrImpl.java:502) 
> at com/ibm/ws/runtime/component/CompositionUnitImpl.stop(CompositionUnitImpl.java:141) 
> at com/ibm/ws/runtime/component/CompositionUnitMgrImpl$2.run(CompositionUnitMgrImpl.java:805) 
> at com/ibm/ws/security/auth/ContextManagerImpl.runAs(ContextManagerImpl.java:5474(Compiled Code)) 
> at com/ibm/ws/security/auth/ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5600(Compiled Code)) 
> at com/ibm/ws/security/core/SecurityContext.runAsSystem(SecurityContext.java:255(Compiled Code)) 
> at com/ibm/ws/runtime/component/CompositionUnitMgrImpl.stopCompositionUnit(CompositionUnitMgrImpl.java:840) 
> at com/ibm/ws/runtime/component/CompositionUnitMgrImpl.stopCompositionUnit(CompositionUnitMgrImpl.java:746) 
> at com/ibm/ws/runtime/component/ApplicationMgrImpl.stopApplication(ApplicationMgrImpl.java:1457) 
> at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) 
> at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76(Compiled Code)) 
> at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43(Compiled Code)) 
> at java/lang/reflect/Method.invoke(Method.java:607(Compiled Code)) 
> at sun/reflect/misc/Trampoline.invoke(MethodUtil.java:75) 
> at sun/reflect/GeneratedMethodAccessor146.invoke(Bytecode PC:40) 
> at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43(Compiled Code)) 
> at java/lang/reflect/Method.invoke(Method.java:607(Compiled Code)) 
> at sun/reflect/misc/MethodUtil.invoke(MethodUtil.java:279) 
> at javax/management/modelmbean/RequiredModelMBean$4.run(RequiredModelMBean.java:1245) 
> at java/security/AccessController.doPrivileged(AccessController.java:300) 
> at java/security/ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76) 
> at javax/management/modelmbean/RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1239) 
> at javax/management/modelmbean/RequiredModelMBean.invoke(RequiredModelMBean.java:1077) 
> at com/sun/jmx/interceptor/DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) 
> at com/sun/jmx/mbeanserver/JmxMBeanServer.invoke(JmxMBeanServer.java:801) 
> at com/ibm/ws/management/AdminServiceImpl$1.run(AdminServiceImpl.java:1335) 
> at com/ibm/ws/security/util/AccessController.doPrivileged(AccessController.java:118(Compiled Code)) 
> at com/ibm/ws/management/AdminServiceImpl.invoke(AdminServiceImpl.java:1228) 
> at com/ibm/ws/management/connector/AdminServiceDelegator.invoke(AdminServiceDelegator.java:181) 
> at com/ibm/ws/management/connector/ipc/CallRouter.route(CallRouter.java:247(Compiled Code)) 
> at com/ibm/ws/management/connector/ipc/IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360(Compiled Code)) 
> at com/ibm/ws/management/connector/ipc/IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602(Compiled Code)) 
> at com/ibm/ws/ssl/channel/impl/SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784(Compiled Code)) 
> at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175(Compiled Code)) 
> at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code)) 
> at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161(Compiled Code)) 
> at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:138(Compiled Code)) 
> at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:204(Compiled Code)) 
> at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:775(Compiled Code)) 
> at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code)) 
> at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1862(Compiled Code))
> {code}
> -----
> h4. Thread 2
> h5. Monitor
> Owns Monitor Lock on org/apache/http/pool/AbstractConnPool$2@0x0000000709E27898
> h5. Java Stack
> {code}
> at sun/misc/Unsafe.park(Native Method) 
> at java/util/concurrent/locks/LockSupport.park(LockSupport.java:186(Compiled Code)) 
> at java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834(Compiled Code)) 
> at java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867(Compiled Code)) 
> at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2047) 
> at org/apache/http/pool/AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:377) 
> at org/apache/http/pool/AbstractConnPool.access$200(AbstractConnPool.java:67) 
> at org/apache/http/pool/AbstractConnPool$2.get(AbstractConnPool.java:243) 
> at org/apache/http/pool/AbstractConnPool$2.get(AbstractConnPool.java:191) 
> at org/apache/http/impl/conn/PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:282) 
> at org/apache/http/impl/conn/PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:269) 
> at org/apache/http/impl/execchain/MainClientExec.execute(MainClientExec.java:191) 
> at org/apache/http/impl/execchain/ProtocolExec.execute(ProtocolExec.java:185) 
> at org/apache/http/impl/execchain/RedirectExec.execute(RedirectExec.java:111) 
> at org/apache/http/impl/client/InternalHttpClient.doExecute(InternalHttpClient.java:185) 
> at org/apache/http/impl/client/CloseableHttpClient.execute(CloseableHttpClient.java:83) 
> at org/apache/http/impl/client/CloseableHttpClient.execute(CloseableHttpClient.java:108) 
> at org/apache/http/impl/client/CloseableHttpClient.execute(CloseableHttpClient.java:56) 
> at com/test/monitor/HttpResourceMonitor.executeHttpGETRequest(HttpResourceMonitor.java:59) 
> at com/test/monitor/HttpResourceMonitor.doMonitor(HttpResourceMonitor.java:47) 
> at com/test/monitor/GenericResourceMonitor.monitor(GenericResourceMonitor.java:34) 
> at sun/reflect/NativeMethodAccessorImpl.invoke0(Native Method) 
> at sun/reflect/NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:76(Compiled Code)) 
> at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43(Compiled Code)) 
> at java/lang/reflect/Method.invoke(Method.java:607(Compiled Code)) 
> at org/springframework/scheduling/support/ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) 
> at org/springframework/scheduling/support/DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) 
> at org/springframework/scheduling/concurrent/ReschedulingRunnable.run(ReschedulingRunnable.java:81) 
> at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:471) 
> at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:334) 
> at java/util/concurrent/FutureTask.run(FutureTask.java:166) 
> at java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) 
> at java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) 
> at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
> at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
> at java/lang/Thread.run(Thread.java:781)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org