commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Basin Ilya <basini...@gmail.com>
Subject Re: Do we really need to read keepalive NOOP responses from FTP server?
Date Fri, 20 Oct 2017 10:06:50 GMT
Hi Sebb

Here are my observations:

1)
When retrieving a file, FTP servers send the transfer status right after they send the EOF
on the data socket, but before the client closes the socket.
This may lead to a race condition:
If we've sent a NOOP not longer than a second ago and want to consume the NOOP reply, we may
receive the transfer status instead. Subsequent call to
completePendingCommand() will consume the NOOP reply.

Maybe it's not a big problem, because we will have either
	226-File successfully transferred
or
	200 NOOP command successful.
as our last message and a high-level FTPClient method will return true.


2) Pure-FTPd:

During a transmission Pure-FTPd replies to NOOPs immediately with:
	500 Unknown command
(perhaps because it only expects ABOR in this state)

At the end of transmission it sends the transmission status (usually 226)


3) BulletProof FTP Server:

During a transmission Pure-FTPd replies to NOOPs immediately with:
	200 NOOP command successful.

If there was at least one NOOP and the command is STOR, the transmission status is not sent.
If there were no NOOPs or the command is RETR, then the status is returned (usually 226)

4) Most other FTP servers:
NOOP replies are delayed. After a transmission completes, servers send 226 first and the NOOP
replies after that.



Here are possible response sequences:

- Pure-FTPd success
	500,500,500,500,226
	500,500,500,226,200 (in case of race condition)

- Pure-FTPd error (RETR data socket closed by client)
	500,500,500,500,150
	500,500,500,150,200 (in case of race condition)



- BulletProof FTP Server STOR, success/error (STOR data socket closed by client)
	200,200,200,200,---
	200,200,200,---,200 (in case of race condition)
        226 (when no NOOPs sent)

- BulletProof FTP Server RETR, success
        226 (when no NOOPs sent)
	200,200,200,200,---
	200,200,200,---,200 (in case of race condition)

- BulletProof FTP Server RETR, error (RETR data socket closed by client)
        426 (when no NOOPs sent)
	200,200,200,200,426
	200,200,200,426,200 (in case of race condition)



- Most other FTP servers success
	226,200,200,200,200 (no race condition possible)

- Most other FTP servers error (RETR data socket closed by client)
	426,200,200,200,200 (no race condition possible)



Conclusion:
If we consume NOOP responses during the transmission, most of the time completePendingCommand()
consumes the right response:
If a server does not support asynchronous NOOPs, then completePendingCommand() consumes the
status and cleanUp() consumes all NOOP responses.
If a server supports asynchronous NOOPs, then all NOOP replies are consumed before completePendingCommand()
is called.
(Except for BulletProof STOR, that forgets to send 226, so completePendingCommand() fails
with SocketTimeoutException)

In case of a race condition (for asyncronous servers):
If transmission succeeds, completePendingCommand() consumes either 226 or "200 NOOP command
successful." and return true in either case.
If transmission fails, Util.copyStream() may throw something. If it doesn't, completePendingCommand()
consumes "200 NOOP command successful." and returns false positive.
This can happen if last data block was sent by client, but never received by server.

If we delay consuming NOOP responses from asynchronous servers, completePendingCommand() always
consumes the first NOOP response.
In case of Pure-FTPd, it is "500 Unknown command", so the method returns false negative.
For other asynchronous servers it is "200 NOOP command successful." and a possible 426 status
is eaten by cleanUp(), so the method returns false positive.

On the other hand, completePendingCommand() won't throw SocketTimeoutException for BulletProof
STOR and besides, for servers that don't support async NOOPs we don't have to
wait in __noop()


I think that what we need to do is to collect (#NOOPs + 1) responses at the end with a read
timeout 10s, filter out 500 and 200 and the remaining status will be 226 or an
error status.




On 19.10.2017 20:46, Basin Ilya wrote:
> Ok, so I found these two that support it:
> - BulletProof FTP Server
> - Pure-FTPd
> 
> Will test further.
> 
> On 17.10.2017 18:29, sebb wrote:
>> On 17 October 2017 at 16:01, Basin Ilya <basinilya@gmail.com> wrote:
>>> Hi sebb
>>>
>>>> No, because some FTP servers *do* support asynchronous control channels.
>>> Do you know any?
>>
>> I cannot remember the name, but I know I came across at least one when testing.
>>
>> But even if there were currently no such servers, AFAIK it is
>> permitted by the RFCs so NET should not assume there are none.
>>
>>>
>>> On 17.10.2017 17:54, sebb wrote:
>>>> On 17 October 2017 at 12:34, Basin Ilya <basinilya@gmail.com> wrote:
>>>>> Hi.
>>>>> I'm using
>>>>> FTPClient.retrieveFileStream()
>>>>>
>>>>> and therefore I need to implement keepalive mechanism by my own.
>>>>
>>>> Not necessarily.
>>>>
>>>> The FTP server does not need the NOOPs.
>>>> They are only needed to deal with routers that detect the inactive
>>>> control channel and decide to drop the connection even though the data
>>>> channel is active.
>>>>
>>>>> I wanted to mimic the implementation from FTPClient.CSL, but then I thought:
>>>>>
>>>>> Most FTP servers don't reply to NOOPs until transmission has finished
and yet, sending NOOPs helps to keep them alive.
>>>>
>>>> No, the FTP servers don't need the NOOPs.
>>>> And some do reply before data transmission completes.
>>>>
>>>>> So we can safely assume that no FTP servers support this and let CSL.cleanUp()
collect all the responses at the end.
>>>>
>>>> No, because some FTP servers *do* support asynchronous control channels.
>>>>
>>>>> My tests show vsftpd 2.2.2 does not support that and you can send up
to 27000 NOOP commands to it until the socket write gets blocked.
>>>>>
>>>>> On the other hand, on most FTP servers for each NOOP keepalive FTPClient
waits for 1000 milliseconds for a reply that never comes. This slows things down a little.
>>>>>
>>>>> What bad thing will happen, if we remove __getReplyNoReport() from FTP.__noop()
and increment notAcked unconditionally?
>>>>
>>>> Try it and see?
>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org


Mime
View raw message