commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Frank <Johannes.Fr...@kisters.de>
Subject Re: [NET] FTP Connection occasionally hangs on listFiles()
Date Fri, 14 Sep 2012 08:56:26 GMT
I have no console to type that in because all our companies java processes 
are indirectly started from a process controller.
However, I have a point where I log out the stack traces to file when 
trying to alter that transfer job.
Here is where the transfer job hangs indefinitely:

2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] 
java.net.SocketInputStream.socketRead0(Native Method)
2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] 
java.net.SocketInputStream.read(SocketInputStream.java:129)
2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] 
sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] 
sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] 
sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
java.io.InputStreamReader.read(InputStreamReader.java:167)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
java.io.BufferedReader.fill(BufferedReader.java:136)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
java.io.BufferedReader.readLine(BufferedReader.java:299)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
java.io.BufferedReader.readLine(BufferedReader.java:362)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
org.apache.commons.net.ftp.FTPFileEntryParserImpl.readNextEntry(FTPFileEntryParserImpl.java:52)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
org.apache.commons.net.ftp.FTPListParseEngine.readStream(FTPListParseEngine.java:133)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
org.apache.commons.net.ftp.FTPListParseEngine.readServerList(FTPListParseEngine.java:102)
2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] 
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3103)
2012-09-14T08:00:41.448Z TRACE [auto.AutoService.handleThreadShutdown] 
org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3072)
2012-09-14T08:00:41.448Z TRACE [auto.AutoService.handleThreadShutdown] 
org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2731)

Note that this stacktrace was made with commons-net 3.1. I already looked 
in the source and it's waiting for a readLine() which never returns. Given 
the fact that the log line "<-- 226
>> Transfer complete. 117 bytes transferred. 7,14 KB/sec." is missing in 
the logs (see my initial post) I assume the library waits for this line to 
come from the server, which for some reason never makes it. What I find 
interesting though is that no timeout seems to interrupt this endless 
waiting. Is this a misconfiguration on my part or a bug in the software? I 
do ftpClient.setSoTimeout(30000);
In my software. Which other Timout might I have to set to cause the 
readLine() to timeout at some point?


Kind regards,
Johannes Frank




From:
Gary Gregory <garydgregory@gmail.com>
To:
Commons Users List <user@commons.apache.org>
Date:
11.09.2012 18:44
Subject:
Re: [NET] FTP Connection occasionally hangs on listFiles()



If you start the JVM from a command line, you can type control-break
to get a thread dump at any time.

Gary

--------------------------------------------------------------------------------------------------------------------------------------------
Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen - Germany
Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns Kisters | Aufsichtsratsvorsitzender:
Dr. Thomas Klevers
Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail: Johannes.Frank@kisters.de | WWW:

--------------------------------------------------------------------------------------------------------------------------------------------
This e-mail may contain confidential and/or privileged information. If you are not the intended
recipient (or have received this e-mail in error) please notify the sender immediately and
destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material
in this e-mail is strictly forbidden.On Sep 11, 2012, at 12:06, sebb <sebbaz@gmail.com>
wrote:

> On 11 September 2012 16:05, Johannes Frank <Johannes.Frank@kisters.de> 
wrote:
>> Hello!
>>
>> As stated in my initial post, the only way of "error log" is the 
missing
>> of the log line stating that the list files command completed "<-- 226
>> Transfer complete. 117 bytes transferred. 7,14 KB/sec."
>>
>> I have to check if I have a data Timeout set, I do definitely have a
>> socket timeout set. Will give the data timout a look.
>> I do not really understand the CommandListener suggestion.
>> Regarding the thread dump I will try if I can find one in the logs we 
had
>> on the customer machine.
>
> Unlikely; a thread dump normally has to be requested manually.
>
> Next time it happens, if possible please generate the thread dump
> before killing the process.
>
> This can be done with jvisualvm which is part of the JDK.
>
>> I am relatively sure it's NOT an issue with the server (as we had the
>> issue with two different servers on two different operating systems),
>
> On the other hand, you say that Commons NET works OK in 99% of cases ...
>
>> which does not mean that the error has to be in NET (could still easily 
be
>> something occult in their network...)
>
>> Thanks by the way for your effort!
>>
>> Regards,
>> Johannes.
>>
>>
>>
>> From:
>> sebb <sebbaz@gmail.com>
>> To:
>> Commons Users List <user@commons.apache.org>
>> Date:
>> 11.09.2012 16:52
>> Subject:
>> Re: [NET] FTP Connection occasionally hangs on listFiles()
>>
>>
>>
>> On 11 September 2012 12:53, Johannes Frank <Johannes.Frank@kisters.de>
>> wrote:
>>> Hello!
>>>
>>> Thank you for your reply. However, I cannot provide such a program 
since
>>> (as said) I cannot reliably reproduce that problem (on 99% of the
>> systems
>>> not at all and on the two mentioned systems only sporadically). From
>> what
>>> I see, simply calling FTPClient.listFiles() on these two systems often
>>> enough will suffice to "reproduce" this problem :-(
>>
>> How are you initialising the FTPClient instance?
>>
>> You could add a command listener using
>> FTPClient#addProtocolCommandListener; if listFiles() returns OK, then
>> delete the output.
>> This might give some useful info.
>>
>> Also, try setting a data timeout using FTPClient#setDataTimeout
>>
>> Is there any error indication in the server logs?
>>
>> Also, can you create a thread dump when the problem occurs?
>> That should show where it is waiting.
>>
>> Without a way to reproduce the issue, it's going to be tricky to debug
>> and fix (if it is a problem with NET rather than a server issue).
>>
>>> Regards,
>>> Johannes Frank
>>>
>>>
>>>
>>>
>>> From:
>>> sebb <sebbaz@gmail.com>
>>> To:
>>> Commons Users List <user@commons.apache.org>
>>> Date:
>>> 11.09.2012 01:00
>>> Subject:
>>> Re: [NET] FTP Connection occasionally hangs on listFiles()
>>>
>>>
>>>
>>> On 10 September 2012 08:09, Johannes Frank <Johannes.Frank@kisters.de>
>>> wrote:
>>>> Hello together.
>>>>
>>>> I have a weird problem which unfortunately leads to production halts 
at
>>>> two of our customers so I urge for a pointer in the right direction
>>> here.
>>>>
>>>> My software periodically detects files on a FTP Server and downloads
>>> them.
>>>> This works fine for some time but quite randomly, out of a suddin 
this
>>>> loop starts to hang indefinitely until manually killed and restarted 
by
>>>> the Administrator.
>>>>
>>>> I narrowed it down so far that I can say that the 
FTPClient.listFiles()
>>>> method hangs and never returns. Here is what the ftp communication 
log
>>>> says on a normal run:
>>>>
>>>> 2012-09-04T12:41:01.130Z DEBUG
>>> [location.FtpLocation.protocolCommandSent]
>>>> Veribox A-Dateien: --> NOOP
>>>> 2012-09-04T12:41:01.132Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
200
>>>> Command okay.
>>>> 2012-09-04T12:41:01.132Z DEBUG
>>> [location.FtpLocation.protocolCommandSent]
>>>> Veribox A-Dateien: --> PASV
>>>> 2012-09-04T12:41:01.140Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
227
>>>> Entering Passive Mode (172,35,1,12,7,70
>>>> )
>>>> 2012-09-04T12:41:01.141Z DEBUG
>>> [location.FtpLocation.protocolCommandSent]
>>>> Veribox A-Dateien: --> LIST *
>>>> 2012-09-04T12:41:01.146Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
150
>>>> Opening ASCII mode data connection for
>>>> /bin/ls.
>>>> 2012-09-04T12:41:01.186Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
226
>>>> Transfer complete. 117 bytes transferre
>>>> d. 7,14 KB/sec.
>>>>
>>>> And here is what it says the moment it hangs up:
>>>>
>>>> 2012-09-04T12:41:51.196Z DEBUG
>>> [location.FtpLocation.protocolCommandSent]
>>>> Veribox A-Dateien: --> NOOP
>>>> 2012-09-04T12:41:51.197Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
200
>>>> Command okay.
>>>> 2012-09-04T12:41:51.198Z DEBUG
>>> [location.FtpLocation.protocolCommandSent]
>>>> Veribox A-Dateien: --> PASV
>>>> 2012-09-04T12:41:51.204Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
227
>>>> Entering Passive Mode (172,35,1,12,7,71
>>>> )
>>>> 2012-09-04T12:41:51.205Z DEBUG
>>> [location.FtpLocation.protocolCommandSent]
>>>> Veribox A-Dateien: --> LIST *
>>>> 2012-09-04T12:41:51.211Z DEBUG
>>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 
150
>>>> Opening ASCII mode data connection for
>>>> /bin/ls.
>>>>
>>>> As you can see, the server answers that it's opening an ascii mode 
data
>>>> connection but then nothing seems to happen.
>>>>
>>>> This is what netstat -np says:
>>>> tcp       91      0 172.30.3.21:39009       172.35.1.12:2730 
CLOSE_WAIT
>>>> 24094/java
>>>> tcp        0      0 172.30.3.21:45273       172.35.1.12:1863
>> ESTABLISHED
>>>> 24094/java
>>>> tcp        0      0 172.30.3.21:56584       172.35.1.12:2299
>> ESTABLISHED
>>>> 24094/java
>>>> tcp        0      0 172.30.3.21:53582       172.35.1.12:2730
>> ESTABLISHED
>>>> 24094/java
>>>> tcp        0      0 172.30.3.21:57945       172.35.1.12:2730
>> ESTABLISHED
>>>> 24094/java
>>>> tcp       95      0 172.30.3.21:39010       172.35.1.12:2730 
CLOSE_WAIT
>>>> 24094/java
>>>>
>>>> 2730 is the control port of the ftp server, 2299, 1863 are data
>>>> connections. Note the non-empty Send-Q for the CLOSE_WAIT 
connections.
>>>>
>>>> The issue I have here seems to be relatively independent of 
everything
>> I
>>>> can come up with.
>>>> We have this problem in both linux (SuSE 11.4 64bit) and Windows XP,
>> The
>>>> FTP server being once a Serv-U under Windows 2003 Standard and vsftp
>>> under
>>>> linux. The java virtual machine is openjdk under linux and sun java 6
>>>> under windows. As far as I can see, no proxies or firewalls are in 
the
>>> way
>>>> on either of the customers systems. I tested with both commons-net 
2.0
>>> and
>>>> 3.1 with no difference in result.
>>>>
>>>> Did anyone on this list ever encounter an issue like this before and 
is
>>>> there a way to circumvent this stalling of the listFiles() call?
>>>
>>> If you can provide a sample program that reproduces the problem, then
>>> please raise a JIRA issue and attach the code and sample output from
>>> the run.
>>> Make sure you enable command logging.
>>>
>>>> Kind regards,
>>>> Johannes Frank
>>>>
>>>
>> 
--------------------------------------------------------------------------------------------------------------------------------------------
>>>> Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen 
-
>>> Germany
>>>> Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns
>>> Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers
>>>> Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail:
>>> Johannes.Frank@kisters.de | WWW:
>>>>
>>>
>> 
--------------------------------------------------------------------------------------------------------------------------------------------
>>>> This e-mail may contain confidential and/or privileged information. 
If
>>> you are not the intended recipient (or have received this e-mail in
>> error)
>>> please notify the sender immediately and destroy this e-mail. Any
>>> unauthorised copying, disclosure or distribution of the material in 
this
>>> e-mail is strictly forbidden.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>>> For additional commands, e-mail: user-help@commons.apache.org
>>>
>>>
>>>
>>>
>>>
>> 
--------------------------------------------------------------------------------------------------------------------------------------------
>>> Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen 
-
>> Germany
>>> Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns
>> Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers
>>> Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail:
>> Johannes.Frank@kisters.de | WWW:
>>>
>> 
--------------------------------------------------------------------------------------------------------------------------------------------
>>> This e-mail may contain confidential and/or privileged information. If
>> you are not the intended recipient (or have received this e-mail in 
error)
>> please notify the sender immediately and destroy this e-mail. Any
>> unauthorised copying, disclosure or distribution of the material in 
this
>> e-mail is strictly forbidden.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>>
>>
>>
>> 
--------------------------------------------------------------------------------------------------------------------------------------------
>> Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen - 
Germany
>> Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns 
Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers
>> Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail: 
Johannes.Frank@kisters.de | WWW:
>> 
--------------------------------------------------------------------------------------------------------------------------------------------
>> This e-mail may contain confidential and/or privileged information. If 
you are not the intended recipient (or have received this e-mail in error) 
please notify the sender immediately and destroy this e-mail. Any 
unauthorised copying, disclosure or distribution of the material in this 
e-mail is strictly forbidden.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message