Mailing-List: contact dev-help@mina.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mina.apache.org
Date: Fri, 26 Sep 2014 13:31:34 +0000 (UTC)
From: =?utf-8?Q?Hugo_Ar=C3=A8s_=28JIRA=29?= <jira@apache.org>
To: dev@mina.apache.org
Message-ID: <JIRA.12740988.1410508817000.130851.1411738294380@Atlassian.JIRA>
In-Reply-To: <JIRA.12740988.1410508817000@Atlassian.JIRA>
References: <JIRA.12740988.1410508817000@Atlassian.JIRA>
 <JIRA.12740988.1410508817720@arcas>
Subject: [jira] [Commented] (SSHD-348) Some SSH threads get blocked in
 Object.wait() method forever
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/SSHD-348?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D141491=
36#comment-14149136 ]=20

Hugo Ar=C3=A8s commented on SSHD-348:
--------------------------------

There are two commit that look suspicious:
8baa36e0e50bbd683dd7f4275d194c94504da4f0 [SSHD-228] Introduce a Closeable i=
nterface and related utilities
18bcdf797dfba5e462413a6ccce9dd6d3da2025b [SSHD-228] Add Closeable to SshCli=
ent, SshService, IoService and forward helpers


> Some SSH threads get blocked in Object.wait() method forever
> ------------------------------------------------------------
>
>                 Key: SSHD-348
>                 URL: https://issues.apache.org/jira/browse/SSHD-348
>             Project: MINA SSHD
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>         Environment: Gerrit Code Review 2.9.1
>            Reporter: David Ostrovsky
>         Attachments: diff
>
>
> This seems to be a regression compared to previous versions (0.6-0 and la=
ter).
> In Gerrit we have SSH commamds that returns immediately and so called str=
eam-events command which keeps connection open until clients disconnect. Ba=
sically for days or weeks. This is used for example to inform CI system (je=
nkins) about events in gerrit, like new patch set upload etc.
> In Gerrit we have configurable "SSH-Stream-Worker" thread pool which is d=
edicated to the mentioned stream-events SSH command. The observed behaviour=
 on latest SSHD release is that after some time all threads are stuck. They=
 aren't stuck at the same time, but one after another untill at some time a=
ll threads are stuck and Gerrit must be restarted. Usually after one week. =
The stack dump of all such stuck thread are the same, see below. If we had =
a patch we could apply it to our production Gerrit instance and try if this=
 helps.
> {code}
> "SSH-Stream-Worker-10" cpu=3D95400.00 [reset 95400.00] ms elapsed=3D14644=
4.30 [reset 146444.30] s allocated=3D5526700000 B (5.15 GB) [reset 55267000=
00 B (5.15 GB)] defined_classes=3D0
> io=3D file i/o: 15622752/0 B, net i/o: 0/0 B, files opened:0, socks opene=
d:0  [reset file i/o: 15622752/0 B, net i/o: 0/0 B, files opened:0, socks o=
pened:0 ]=20
> prio=3D10 tid=3D0x00007f54514df800 nid=3D0x1c71 / 7281  pthread-id=3D1399=
99281374976 in Object.wait()  [_thread_blocked (_at_safepoint), stack(0x000=
07f541f5f6000,0x00007f541f6f7000)] [0x00007f541f6f5000]
>    java.lang.Thread.State: WAITING (on object monitor)
> =09at java.lang.Object.wait(J)V(Native Method)
> =09- waiting on <0x00007f553aa530d0> (a org.apache.sshd.common.channel.Wi=
ndow)
> =09at java.lang.Object.wait()V(Object.java:503)
> =09at org.apache.sshd.common.channel.Window.waitForSpace()I(Window.java:1=
48)
> =09- locked <0x00007f553aa530d0> (a org.apache.sshd.common.channel.Window=
)
> =09at org.apache.sshd.common.channel.ChannelOutputStream.flush()V(Channel=
OutputStream.java:116)
> =09- locked <0x00007f553aa55060> (a org.apache.sshd.common.channel.Channe=
lOutputStream)
> =09at org.apache.sshd.common.channel.ChannelOutputStream.write([BII)V(Cha=
nnelOutputStream.java:84)
> =09- locked <0x00007f553aa55060> (a org.apache.sshd.common.channel.Channe=
lOutputStream)
> =09at sun.nio.cs.StreamEncoder.writeBytes()V(StreamEncoder.java:221)
> =09at sun.nio.cs.StreamEncoder.implFlushBuffer()V(StreamEncoder.java:291)
> =09at sun.nio.cs.StreamEncoder.implFlush()V(StreamEncoder.java:295)
> =09at sun.nio.cs.StreamEncoder.flush()V(StreamEncoder.java:141)
> =09- locked <0x00007f553aa7e258> (a java.io.OutputStreamWriter)
> =09at java.io.OutputStreamWriter.flush()V(OutputStreamWriter.java:229)
> =09at java.io.BufferedWriter.flush()V(BufferedWriter.java:254)
> =09- locked <0x00007f553aa7e258> (a java.io.OutputStreamWriter)
> =09at java.io.PrintWriter.flush()V(PrintWriter.java:320)
> =09- locked <0x00007f553aa7e210> (a java.io.BufferedWriter)
> =09at java.io.PrintWriter.checkError()Z(PrintWriter.java:357)
> =09at com.google.gerrit.sshd.commands.StreamEvents.writeEvents()V(StreamE=
vents.java:186)
> =09at com.google.gerrit.sshd.commands.StreamEvents.access$100(Lcom/google=
/gerrit/sshd/commands/StreamEvents;)V(StreamEvents.java:43)
> =09at com.google.gerrit.sshd.commands.StreamEvents$3.run()V(StreamEvents.=
java:82)
> =09at java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Obj=
ect;(Executors.java:471)
> =09at java.util.concurrent.FutureTask.run()V(FutureTask.java:262)
> =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas=
k.access$201(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFut=
ureTask;)V(ScheduledThreadPoolExecutor.java:178)
> =09at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTas=
k.run()V(ScheduledThreadPoolExecutor.java:292)
> =09at com.google.gerrit.server.git.WorkQueue$Task.run()V(WorkQueue.java:3=
64)
> =09at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concur=
rent/ThreadPoolExecutor$Worker;)V(ThreadPoolExecutor.java:1145)
> =09at java.util.concurrent.ThreadPoolExecutor$Worker.run()V(ThreadPoolExe=
cutor.java:615)
> =09at java.lang.Thread.run()V(Thread.java:812)
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)