commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Seganti, Michael (DOF)" <Segan...@finance.nyc.gov>
Subject RE: [net] FTPClient Transfer succeeds but fails in cleanUp
Date Fri, 18 Nov 2016 19:46:31 GMT
Same code on two different source hosts going to the same destination fails on occasion.  I've
never had an issue with any other destination hosts using this code (5 years?). 

I had to add a bit to get the example code to work (pbsz and prot, as well as site commands),
but I was able to successfully get the file transferred. 

A little background on my issue, the mainframe I'm sending to is fairly saturated, and I suspected
the issue was caused by that.  

I started playing around with the control keepalive timeout's, which my understanding is,
that a NOOP gets sent over the control connection every keepalive timeout period. 

In my case, I override it by default to 4:30.  I thought maybe the control connection was
closing (timed out), and I wasn't receiving the successful reply, so I dropped it to 60 seconds.


I have one 2GB file which I can get to fail consistently using my code.  When I set the keepAliveTimout
(-k 60) using the FTPClientExample, it also fails in the same way.  

It's odd that the protocol listener doesn't print the result, even though the file transmission
completed.  

Command: 

java -cp commons-net-3.5.jar;. FTPClientExample -p false -# -s -k 60 mvsp.xxx:2121 C836PTS
password 'A836.ORG.CAMA2.FULL' d:\LargeExtracts\full.txt

FTPClientExample Log Snippet: 

#####################java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at sun.security.ssl.InputRecord.readFully(Unknown Source)
        at sun.security.ssl.InputRecord.read(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
        at sun.security.ssl.AppInputStream.read(Unknown Source)
        at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
        at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
        at sun.nio.cs.StreamDecoder.read(Unknown Source)
        at java.io.InputStreamReader.read(Unknown Source)
        at java.io.BufferedReader.fill(Unknown Source)
        at java.io.BufferedReader.readLine(Unknown Source)
        at java.io.BufferedReader.readLine(Unknown Source)
        at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
        at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:303)
        at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3839)
        at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:695)
        at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:643)
        at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2034)
        at FTPClientExample.main(FTPClientExample.java:415)


-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com] 
Sent: Thursday, November 17, 2016 2:49 PM
To: Commons Users List
Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp

Does the same code fail when sending the same file to a different host?

Have you tried using the example utility?

http://commons.apache.org/proper/commons-net/#Examples

This exercises most of the FTP methods.


On 17 November 2016 at 18:01, Seganti, Michael (DOF) <SegantiM@finance.nyc.gov> wrote:
> I just updated to 3.5, and this file is consistently failing.
>
> I no longer receive the
>
> 250 Transfer completed successfully.
> 200 OK
>
> Replies.
>
> Log Snippet:
>
> 2016-11-17 12:56:33.870 - 100 Percent Complete, Total Bytes 
> Transferred: 2101359588
>
>         File Transfer Failed at: 2016-11-17 12:56:38.885 Error 
> Occurred Transmitting File to Remote System, aborting...
>
> Read timed out
> java.net.SocketTimeoutException: Read timed out
>         at java.net.SocketInputStream.socketRead0(Native Method)
>         at java.net.SocketInputStream.read(Unknown Source)
>         at java.net.SocketInputStream.read(Unknown Source)
>         at sun.security.ssl.InputRecord.readFully(Unknown Source)
>         at sun.security.ssl.InputRecord.read(Unknown Source)
>         at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
>         at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
>         at sun.security.ssl.AppInputStream.read(Unknown Source)
>         at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
>         at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
>         at sun.nio.cs.StreamDecoder.read(Unknown Source)
>         at java.io.InputStreamReader.read(Unknown Source)
>         at java.io.BufferedReader.fill(Unknown Source)
>         at java.io.BufferedReader.readLine(Unknown Source)
>         at java.io.BufferedReader.readLine(Unknown Source)
>         at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:313)
>         at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.java:303)
>         at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(FTPClient.java:3839)
>         at org.apache.commons.net.ftp.FTPClient._storeFile(FTPClient.java:695)
>         at org.apache.commons.net.ftp.FTPClient.__storeFile(FTPClient.java:643)
>         at org.apache.commons.net.ftp.FTPClient.storeFile(FTPClient.java:2034)
>         at gov.nyc.ftputils.FTPConnection.sendFile(FTPConnection.java:695)
>         at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:54)
>         at gov.nyc.ftputils.FTPPropertyProcessor.putFile(FTPPropertyProcessor.java:96)
>         at gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(FTPPropertyProcessor.java:126)
>         at gov.nyc.ftputils.FTPClientUtil.main(FTPClientUtil.java:378)
> FTPClientUtil has Failed
> QUIT
> 221 Quit command received. Goodbye.
>
>
> -----Original Message-----
> From: Seganti, Michael (DOF) [mailto:SegantiM@finance.nyc.gov]
> Sent: Thursday, November 17, 2016 12:34 PM
> To: Commons Users List
> Subject: RE: [net] FTPClient Transfer succeeds but fails in cleanUp
>
> I need to do a quick test prior to pushing to production, but yes, that's my next step.
>
> I didn't see anything that stood out in the release notes.  It would make life easier
if this was reproduceable...
>
> -----Original Message-----
> From: Gary Gregory [mailto:garydgregory@gmail.com]
> Sent: Thursday, November 17, 2016 12:31 PM
> To: Commons Users List
> Subject: Re: [net] FTPClient Transfer succeeds but fails in cleanUp
>
> Can you try version 3.5?
>
> Gary
>
> On Thu, Nov 17, 2016 at 9:19 AM, Seganti, Michael (DOF) < SegantiM@finance.nyc.gov>
wrote:
>
>> I'm having a sporadic issue transferring files to a Mainframe, was 
>> wondering if anyone else has run into this.  I'm trying to figure out 
>> if this is a Mainframe issue, or a net commons issue.
>>
>> I'm using Java 1.7 and commons-net-3.3.jar
>>
>> The file transfer completes, but it looks like either the 200 OK is 
>> sent over the wrong connection, or is not properly being parsed and 
>> is causing the cleanup to fail.
>>
>> According to the reply: IBM FTP CS V2R2 is the version where the 
>> issue occurs.
>>
>> I have a second mainframe that is running IBM FTP CS V1R13 that 
>> doesn't seem to have this issue
>>
>> In a normal transfer, you would see:
>>
>> 250 Transfer completed successfully.
>>        File Transfer Completed Successfully at: 2016-11-17
>> 04:30:51.569 QUIT
>> 221 Quit command received. Goodbye.
>>
>> This is what we are getting on the failures:
>>
>> 250 Transfer completed successfully.
>> 200 OK
>>
>>        File Transfer Failed at: 2016-11-17 04:36:50.246 Error 
>> Occurred Transmitting File to Remote System, aborting...
>>
>> Read timed out
>>
>> Here's the full log from the failed transfer:
>>
>> Using FTPS Connection with Protocol: TLS and Explicit Security 
>> Request
>> 220-FTPSECR1 IBM FTP CS V2R2 at MVSP.XXX, 09:27:29 on 2016-11-17.
>> 220 Connection will close if idle for more than 5 minutes.
>> AUTH TLS
>> 234 Security environment established - ready for negotiation 
>> Connected to mvsp.xxx on xxxx PBSZ 0
>> 200 Protection buffer size accepted
>> PROT P
>> 200 Data connection protection set to private USER *******
>> 331 Send password please.
>> PASS *******
>> 230 C836PTS is logged on.  Working directory is "C836PTS.".
>> PWD
>> 257 "'C836PTS.'" is working directory.
>>
>> Using a Packet Size of: 4096
>> Control KeepAliveTimeOut: 60
>> Control KeepAliveReplyTimeOut: 5000
>> FEAT
>> 211- Extensions supported
>> AUTH TLS
>> PBSZ
>> PROT
>> 211 End
>>
>> ...FTPClientUtil Connected Successfully!
>> SITE TRAIL
>> 200 SITE command was accepted
>> SITE CYLINDERS PRIMARY=2100 SECONDARY=250
>> 200 SITE command was accepted
>> SITE RECFM=FB LRECL=1915 BLKSIZE=30640
>> 200 SITE command was accepted
>> SITE CONDDISP=delete
>> 200 SITE command was accepted
>>
>>
>> Sending:
>>                 Local: d:\LargeExtracts\full.txt
>>                 Remote: 'A836.ORG.CAMA2.FULL'
>>                 Starting at: 2016-11-17 09:27:34.798 Total Bytes To
>> Send: 2101359588 PASV
>> 227 Entering Passive Mode (xx,xx,x,xx,xx,xx) STOR 
>> 'A836.ORG.CAMA2.FULL'
>> 125 Storing data set A836.ORG.CAMA2.FULL
>> 2016-11-17 09:27:47.721 - 5 Percent Complete, Total Bytes Transferred:
>> 105070592
>> 2016-11-17 09:28:01.190 - 10 Percent Complete, Total Bytes Transferred:
>> 210137088
>> ...
>> 2016-11-17 09:32:21.127 - 100 Percent Complete, Total Bytes Transferred:
>> 2101359588
>> 250 Transfer completed successfully.
>> 200 OK
>>
>>                 File Transfer Failed at: 2016-11-17 09:32:26.127 
>> Error Occurred Transmitting File to Remote System, aborting...
>>
>> Read timed out
>> java.net.SocketTimeoutException: Read timed out
>>                 at java.net.SocketInputStream.socketRead0(Native Method)
>>                 at java.net.SocketInputStream.read(Unknown Source)
>>                 at java.net.SocketInputStream.read(Unknown Source)
>>                 at sun.security.ssl.InputRecord.readFully(Unknown Source)
>>                 at sun.security.ssl.InputRecord.read(Unknown Source)
>>                 at sun.security.ssl.SSLSocketImpl.readRecord(Unknown
>> Source)
>>                 at
>> sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown
>> Source)
>>                 at sun.security.ssl.AppInputStream.read(Unknown Source)
>>                 at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
>>                 at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
>>                 at sun.nio.cs.StreamDecoder.read(Unknown Source)
>>                 at java.io.InputStreamReader.read(Unknown Source)
>>                 at java.io.BufferedReader.fill(Unknown Source)
>>                 at java.io.BufferedReader.readLine(Unknown Source)
>>                 at java.io.BufferedReader.readLine(Unknown Source)
>>                 at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:314)
>>                 at org.apache.commons.net.ftp.FTP.__getReplyNoReport(FTP.
>> java:304)
>>                 at org.apache.commons.net.ftp.FTPClient$CSL.cleanUp(
>> FTPClient.java:3697)
>>                 at org.apache.commons.net.ftp.FTPClient._storeFile(
>> FTPClient.java:669)
>>                 at org.apache.commons.net.ftp.FTPClient.__storeFile(
>> FTPClient.java:624)
>>                 at org.apache.commons.net.ftp.
>> FTPClient.storeFile(FTPClient.java:1976)
>>                 at gov.nyc.ftputils.FTPConnection.sendFile(
>> FTPConnection.java:695)
>>                 at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
>> FTPPropertyProcessor.java:54)
>>                 at gov.nyc.ftputils.FTPPropertyProcessor.putFile(
>> FTPPropertyProcessor.java:96)
>>                 at
>> gov.nyc.ftputils.FTPPropertyProcessor.processFileProps(
>> FTPPropertyProcessor.java:126)
>>                 at gov.nyc.ftputils.FTPClientUtil.main(
>> FTPClientUtil.java:378)
>> FTPClientUtil has Failed
>> QUIT
>> 221 Quit command received. Goodbye.
>>
>>
>> Here's a code snippet of the transfer code:
>>
>> try{
>>             if (trace){
>>
>> System.out.println("Total Bytes To Send: " + localFile.length());
>>                                                 CopyStreamListener 
>> listener = new ProgressStreamListener(localFile.length());
>>
>> ftp.setCopyStreamListener( listener);
>>                                 }
>>
>>                 InputStream input = new FileInputStream(localFileName);
>>                 boolean result = ftp.storeFile(tempRemoteFileName, input);
>>                 input.close();
>>
>>                 if (result){
>>                                 System.out.println("\tFile Transfer 
>> Completed Successfully at: " + sdf.format(Calendar.
>> getInstance().getTime()));
>>                                 ...
>>                 }
>>                 else
>>                                 System.out.println("\tFile Transfer 
>> Failed
>> at: " + sdf.format(Calendar.getInstance().getTime()));
>>
>>                 return result;
>> }catch (CopyStreamException cse){
>>                 System.err.println("\n\tFile Transfer Failed at: " +
>> sdf.format(Calendar.getInstance().getTime()) + ", Failure occurred 
>> after " + cse.getTotalBytesTransferred() + " Bytes Transferred");
>>                 System.err.println("Error Occurred Transmitting File 
>> to Remote System, aborting...\n");
>>                 cse.printStackTrace(System.err);
>>                 System.err.println("\n\nIOException Stack Trace that 
>> Caused the Error:\n");
>>                 cse.getIOException().printStackTrace(System.err);
>>                 return false;
>> }catch (IOException ioe){
>> System.err.println("\n\tFile Transfer Failed at: " + sdf.format(Calendar.
>> getInstance().getTime()));
>>                 System.err.println("Error Occurred Transmitting File 
>> to Remote System, aborting...\n");
>>                 System.err.println(ioe.getMessage());
>>                 ioe.printStackTrace(System.err);
>>                 return false;
>> }catch (Exception e){
>>                 System.err.println("\n\tFile Transfer Failed at: " + 
>> sdf.format(Calendar.getInstance().getTime()));
>>                 System.err.println("Error Occurred Transmitting File 
>> to Remote System, aborting...\n");
>>                 e.printStackTrace(System.err);
>>                 return false;
>> }
>> }
>>
>>
>> Thanks,
>>
>> Mike...
>>
>
>
>
> --
> E-Mail: garydgregory@gmail.com | ggregory@apache.org Java Persistence 
> with Hibernate, Second Edition 
> <https://www.amazon.com/gp/product/1617290459/ref=as_li_tl?ie=UTF8&cam
> p=1789&creative=9325&creativeASIN=1617290459&linkCode=as2&tag=garygreg
> ory-20&linkId=cadb800f39946ec62ea2b1af9fe6a2b8>
>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=
> 1617290459>
> JUnit in Action, Second Edition
> <https://www.amazon.com/gp/product/1935182021/ref=as_li_tl?ie=UTF8&cam
> p=1789&creative=9325&creativeASIN=1935182021&linkCode=as2&tag=garygreg
> ory-20&linkId=31ecd1f6b6d1eaf8886ac902a24de418%22>
>
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=
> 1935182021>
> Spring Batch in Action
> <https://www.amazon.com/gp/product/1935182951/ref=as_li_tl?ie=UTF8&cam
> p=1789&creative=9325&creativeASIN=1935182951&linkCode=%7B%7BlinkCode%7
> D%7D&tag=garygregory-20&linkId=%7B%7Blink_id%7D%7D%22%3ESpring+Batch+i
> n+Action> 
> <http:////ir-na.amazon-adsystem.com/e/ir?t=garygregory-20&l=am2&o=1&a=
> 1935182951>
> Blog: http://garygregory.wordpress.com
> Home: http://garygregory.com/
> Tweet! http://twitter.com/GaryGregory

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org

Mime
View raw message