mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Goldstein Lyor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SSHD-844) Deadlock detected in Apache SSHD client code
Date Tue, 18 Sep 2018 13:18:00 GMT

    [ https://issues.apache.org/jira/browse/SSHD-844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619092#comment-16619092
] 

Goldstein Lyor commented on SSHD-844:
-------------------------------------

{quote}
Do you think acting on thread pool may mitigate this issue or something else ?
{quote}
To some extent, but I doubt it is a good solution. I recommend you consider using a throttling
mechanism to somehow moderate the data transfer. You can either do it on your part of the
code that generates the data or use a more generic [throttling mechanism as described in the
documentation|https://github.com/apache/mina-sshd/#data-streams-sizing-consideration]. If
you do successfully implement a generic throttling mechanism, we would really appreciate it
if you would consider contributing the code to the MINA SSHD project.

> Deadlock detected in Apache SSHD client code
> --------------------------------------------
>
>                 Key: SSHD-844
>                 URL: https://issues.apache.org/jira/browse/SSHD-844
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 2.0.1
>         Environment: AIX 7.2
> java version "1.8.0_151"
> Java(TM) SE Runtime Environment (build 8.0.5.7 - pap6480sr5fp7-20171216_01(SR5 FP7))
> IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64 Compressed References 20171215_373586 (JIT
enabled, AOT enabled)
> OpenJ9   - 5aa401f
> OMR      - 101e793
> IBM      - b4a79bf)
> JCL - 20171214_01 based on Oracle jdk8u151-b12
>            Reporter: Roberto Deandrea
>            Priority: Critical
>         Attachments: javacore.20180913.153332.14221746.0001.txt, jca457.jar
>
>
> I found a deadlock in Apache SSHD client code running integration test for a local port
forwarding application. Attached the IBM javacore with information about the deadlock.
> Please run $java -jar jca457.jar to load the javacore file and display thread stack traces.
> [^javacore.20180913.153332.14221746.0001.txt]
> [^jca457.jar]
>  
> There are 2 threads in DEADLOCK belonging to client pool's threads, that blocked the
other pool threads from working.
> The locks involved are LOCK1 (object ChannelOutputStream) and LOCK2 Object encodeLock
of AbstractSession class.
>  
> The below thread is waiting on LOCK2 and owns LOCK1 (that is ChannelOutputStream object)
> The thread sshd-SshClient[61e9f863]-nio2-thread-6 has the following stack trace :
>  
> at org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1189(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled
Code)) 
> at org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled
Code)) 
> at org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled
Code)) 
> at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.messageReceived(DefaultForwardingFilter.java:1063(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode
PC:12(Compiled Code)) 
> at java/security/AccessController.doPrivileged(AccessController.java:638(Compiled Code))

> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled
Code)) 
> at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code)) 
> at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code)) 
> at sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled
Code)) 
> at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled
Code)) 
> at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
> at java/lang/Thread.run(Thread.java:811)
>  
> This thread is waiting for LOCK1 and owns LOCK2.
> The thread sshd-SshClient[61e9f863]-nio2-thread-10 has the following stack trace:
>  
> at org/apache/sshd/common/channel/ChannelOutputStream.close(ChannelOutputStream.java:249(Compiled
Code)) 
> at org/apache/sshd/common/util/io/IoUtils.closeQuietly(IoUtils.java:151(Compiled Code))

> at org/apache/sshd/client/channel/AbstractClientChannel.lambda$getInnerCloseable$1(AbstractClientChannel.java:203(Compiled
Code)) 
> at org/apache/sshd/client/channel/AbstractClientChannel$$Lambda$314/00000000252C5370.run(Bytecode
PC:4(Compiled Code)) 
> at org/apache/sshd/common/util/closeable/Builder$1.doClose(Builder.java:47(Compiled Code))

> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code)) 
> at org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code)) 
> at org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseGracefully(AbstractInnerCloseable.java:40(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:98(Compiled
Code)) 
> at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.sessionClosed(DefaultForwardingFilter.java:1043(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.doCloseImmediately(Nio2Session.java:266(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code)) 
> at org/apache/sshd/common/future/DefaultSshFuture.addListener(DefaultSshFuture.java:167(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:57(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/ParallelCloseable.doClose(ParallelCloseable.java:65(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:56(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable$1.operationComplete(SequentialCloseable.java:45(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SequentialCloseable.doClose(SequentialCloseable.java:69(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/SimpleCloseable.close(SimpleCloseable.java:63(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46(Compiled
Code)) 
> at org/apache/sshd/common/util/closeable/AbstractCloseable.close(AbstractCloseable.java:83(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.exceptionCaught(AbstractSession.java:988)

> at org/apache/sshd/client/session/ClientSessionImpl.exceptionCaught(ClientSessionImpl.java:121)

> at org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:53)

> at org/apache/sshd/common/io/nio2/Nio2Session.exceptionCaught(Nio2Session.java:184(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.handleWriteCycleFailure(Nio2Session.java:460)

> at org/apache/sshd/common/io/nio2/Nio2Session$2.onFailed(Nio2Session.java:415) 
> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$324/00000000252CCA50.run(Bytecode
PC:12(Compiled Code)) 
> at java/security/AccessController.doPrivileged(AccessController.java:638(Compiled Code))

> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45(Compiled
Code)) 
> at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:140(Compiled Code)) 
> at sun/nio/ch/Invoker.invokeDirect(Invoker.java:169(Compiled Code)) 
> at sun/nio/ch/UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:749(Compiled
Code)) 
> at sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:394(Compiled
Code)) 
> at sun/nio/ch/AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:411(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.doWriteCycle(Nio2Session.java:401(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.startWriting(Nio2Session.java:386(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.writePacket(Nio2Session.java:167(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1196(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1137(Compiled
Code)) 
> at org/apache/sshd/common/channel/AbstractChannel.writePacket(AbstractChannel.java:773(Compiled
Code)) 
> at org/apache/sshd/common/channel/ChannelOutputStream.flush(ChannelOutputStream.java:227(Compiled
Code)) 
> at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler.lambda$messageReceived$1(DefaultForwardingFilter.java:1069)

> at org/apache/sshd/common/forward/DefaultForwardingFilter$StaticIoHandler$$Lambda$327/00000000246D5DB0.operationComplete(Bytecode
PC:20) 
> at org/apache/sshd/common/future/AbstractSshFuture.notifyListener(AbstractSshFuture.java:159(Compiled
Code)) 
> at org/apache/sshd/common/future/DefaultSshFuture.notifyListeners(DefaultSshFuture.java:217(Compiled
Code)) 
> at org/apache/sshd/common/future/DefaultSshFuture.setValue(DefaultSshFuture.java:112(Compiled
Code)) 
> at org/apache/sshd/client/future/DefaultOpenFuture.setOpened(DefaultOpenFuture.java:65(Compiled
Code)) 
> at org/apache/sshd/client/channel/AbstractClientChannel.handleOpenSuccess(AbstractClientChannel.java:352(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractConnectionService.channelOpenConfirmation(AbstractConnectionService.java:423(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractConnectionService.process(AbstractConnectionService.java:333(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.doHandleMessage(AbstractSession.java:626(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.handleMessage(AbstractSession.java:559(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.decode(AbstractSession.java:1542(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSession.messageReceived(AbstractSession.java:520(Compiled
Code)) 
> at org/apache/sshd/common/session/helpers/AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session.handleReadCycleCompletion(Nio2Session.java:339(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:318(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2Session$1.onCompleted(Nio2Session.java:315(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38(Compiled
Code)) 
> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler$$Lambda$254/0000000023F76250.run(Bytecode
PC:12(Compiled Code)) 
> at java/security/AccessController.doPrivileged(AccessController.java:638(Compiled Code))

> at org/apache/sshd/common/io/nio2/Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37(Compiled
Code)) 
> at sun/nio/ch/Invoker.invokeUnchecked(Invoker.java:138(Compiled Code)) 
> at sun/nio/ch/Invoker$2.run(Invoker.java:230(Compiled Code)) 
> at sun/nio/ch/AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:124(Compiled
Code)) 
> at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160(Compiled
Code)) 
> at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 
> at java/lang/Thread.run(Thread.java:811)
>  
> Can you find a solution to avoid this deadlock ?
> Thanks in advance
> Roberto
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message