camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: "Unconnected sockets not implemented" exception in camel-ftp when using ftps
Date Mon, 21 Jun 2010 07:28:14 GMT
On Mon, Jun 21, 2010 at 9:11 AM, Bengt Rodehav <bengt@rodehav.com> wrote:
> OK
>

Bengt could you try changing camel-ftp in the RemoteFileProducer
    public class RemoteFileProducer<T> extends GenericFileProducer<T>
implements ServicePoolAware {
And remove the implements ServicePoolAware which should cause Camel to
not pool it and thus you get a new fresh instance each time.

Then try testing with that to see if that fixes the re-connect issue.

What we can then do is to add some nice option in the endpoint URI so
you can configure the pooling behavior.


> 2010/6/21 Claus Ibsen <claus.ibsen@gmail.com>
>
>> On Mon, Jun 21, 2010 at 8:52 AM, Bengt Rodehav <bengt@rodehav.com> wrote:
>> > I have now created a JIRA in commons-net for this issue:
>> >
>> > https://issues.apache.org/jira/browse/NET-327
>> >
>>
>> Good
>>
>> > <https://issues.apache.org/jira/browse/NET-327>However, I still think
we
>> > must fix this in camel-ftp by some kind of work-around (maybe similar to
>> the
>> > one I proposed in https://issues.apache.org/activemq/browse/CAMEL-2829).
>> > Have you had a chance to look at it Claus?
>> >
>>
>> No sorry I am to busy with other issues but we got the ticket so we
>> will look into it before 2.4 is cut.
>>
>>
>> > /Bengt
>> >
>> > 2010/6/18 Bengt Rodehav <bengt@rodehav.com>
>> >
>> >> JIRA created:
>> >>
>> >> https://issues.apache.org/activemq/browse/CAMEL-2829
>> >>
>> >> <https://issues.apache.org/activemq/browse/CAMEL-2829>/Bengt
>> >>
>> >> 2010/6/18 Bengt Rodehav <bengt@rodehav.com>
>> >>
>> >> Claus,
>> >>>
>> >>> I'm not sure what you mean by:
>> >>>
>> >>> "I think we should add an option on the endpoint to allow people to
>> >>> turn this on/off.
>> >>> Some would like to reuse existing connections to avoid the connect ->
>> >>> upload -> disconnect cycle when they upload many files etc."
>> >>>
>> >>> IMO we would not disconnect more often than today. We would just change
>> >>> the connect() method in FtpsOperations so that it creates a new
>> instance of
>> >>> FTPSClient prior to a connection attempt. I assume that since the
>> connect()
>> >>> method is called in the first place, we are not currently connected
and
>> >>> there is no way to reuse an existing connection. Am I right?
>> >>>
>> >>> I have made some modifications in camel-ftp along these lines. I will
>> >>> create a JIRA ticket for this bug and attach my changes as diff files
>> to the
>> >>> issue. I just tested my changes and they do solve the problem I have
>> >>> encountered but I haven't tested whether I have introduced any new
>> problems
>> >>> or not.
>> >>>
>> >>> Anyway, this is what I did:
>> >>>
>> >>> *FtpsEndpoint*
>> >>> I added the following members to keep track of the configuration of
the
>> >>> FTPSClient:
>> >>>
>> >>>   private boolean isNeedClientAuth = false;
>> >>>   private KeyManager keyManager;
>> >>>   private TrustManager trustManager;
>> >>>
>> >>> I modified the createFtpClient() method so that it doesn't actually
>> create
>> >>> a FTPSClient but only calculates and stores the configuration in the
>> above
>> >>> members. I also renamed it to createFtpsConfiguration() to reflect
>> that.
>> >>>
>> >>> The createRemoteFileOperations() method was also changed accordingly.
>> It
>> >>> no longer interacts directly with an FTPSClient (since it is not
>> created
>> >>> yet).
>> >>>
>> >>> *FtpsOperations*
>> >>> I added the private createFtpsClient() method that takes care of
>> creating
>> >>> an FTPSClient from configuration stored in the FtpsEndpoint. The
>> >>> createFtpsClient() is called in the connect() method just before the
>> call to
>> >>> the "real" connect() method in the superclass (FtpOperations).
>> >>>
>> >>> I changed the constructor so that it no longer takes a client as an
>> >>> argument. I send null as client to the super class constructor. This
is
>> of
>> >>> course a major change in behavior that you need to take a look at it.
>> >>> Neither FtpsOperations nor its super class FtpOperations can no longer
>> >>> assume that a client exists until the first connection attempt has been
>> >>> made.
>> >>>
>> >>> *FtpOperations*
>> >>> I had to make the "client" member non-final since I now set it on every
>> >>> call to the connect() method in FtpsOperations.
>> >>>
>> >>>
>> >>> BTW, I agree with you that the SFTPClient in commons-net is flawed.
It
>> >>> should provide a way to re-initialize itself so that a new connect()
>> could
>> >>> be made. The FTPClient seems to work that way and it also provides a
>> >>> disconnect() method. If they do not intend to support this, then they
>> should
>> >>> at least state, as part of the documentation, that the SFTPClient
>> cannot be
>> >>> reused across multiple connections. I've posted about this on their
>> mailing
>> >>> list but hasn't got any reply yet. Meanwhile the above can be regarded
>> as a
>> >>> (complicated...) work-around.
>> >>>
>> >>> /Bengt
>> >>>
>> >>>
>> >>> 2010/6/18 Claus Ibsen <claus.ibsen@gmail.com>
>> >>>
>> >>> On Thu, Jun 17, 2010 at 11:56 PM, Bengt Rodehav <bengt@rodehav.com>
>> >>>> wrote:
>> >>>> > Claus,
>> >>>> >
>> >>>> > Having looked a bit more at the commons-net code it's becoming
clear
>> to
>> >>>> me
>> >>>> > that you probably can't just reconnect an FTPSClient once it
has
>> >>>> entered
>> >>>> > secure communication mode. I can't find a way to re-initialise
an
>> >>>> > FTPSClient. I tried the disconnect() method (in the super class
>> >>>> FTPClient)
>> >>>> > but it didn't get the FTPSClient out of secure communication
mode.
>> >>>> >
>> >>>> > To be safe I think we should always start with a newly instantiated
>> >>>> > FTPSClient when we try to connect. Today the FTPSClient is
created
>> by
>> >>>> the
>> >>>> > FtpsEndpoint. I don't think it has to. It could just get all
the
>> righ
>> >>>> > configuration parameters and pass it to FtpsOperations which
would
>> then
>> >>>> > instantiate a new FTPSClient at every connect attempt.
>> >>>> >
>> >>>> > If we do it this way for ftps I guess it would make sense with
the
>> same
>> >>>> > setup for plain ftp.
>> >>>> >
>> >>>> > What do you think?
>> >>>> >
>> >>>>
>> >>>> I think we should add an option on the endpoint to allow people
to
>> >>>> turn this on/off.
>> >>>> Some would like to reuse existing connections to avoid the connect
->
>> >>>> upload -> disconnect cycle when they upload many files etc.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> And frankly I can't see the problem why a secure connection cannot
>> >>>> self heal and be able to re-establish itself. After all FTPSClient
got
>> >>>> all the connection information. And hence I would consider it a
>> >>>> missing feature / glitch in the FTP library.
>> >>>>
>> >>>> Imagine if using secure connections for JMS brokers which couldn't
>> >>>> self heal just because a network outage. That would cause a lot
of
>> >>>> pain for clients having to re-start.
>> >>>>
>> >>>>
>> >>>> > /Bengt
>> >>>> >
>> >>>> > 2010/6/17 Bengt Rodehav <bengt@rodehav.com>
>> >>>> >
>> >>>> >> Claus,
>> >>>> >>
>> >>>> >> Yeah, maybe this is a commons-net issue. As always I find
it
>> >>>> surprising
>> >>>> >> that no one else has encountered this problem.
>> >>>> >>
>> >>>> >> As I started writing an email on the commons user list,
I took
>> another
>> >>>> look
>> >>>> >> at the code in commons-net. It turns out that the FTPSSocketFactory
>> >>>> class
>> >>>> >> actually does override the createSocket() methods in SocketFactory.
>> A
>> >>>> >> similar work-around that I was talking about but on the
factory
>> class
>> >>>> >> instead of in the SocketClient class. FTPSSocketFactory
delegates
>> the
>> >>>> >> createSocket() calls to its SSLContect's socket factory.
>> >>>> >>
>> >>>> >> So why then are we initially able to connect but not after
we have
>> >>>> called
>> >>>> >> execProt() and thus changed the connection factory?
>> >>>> >>
>> >>>> >> Moving back to the FTPSClient class now. When the initial
>> connection
>> >>>> is up,
>> >>>> >> the _connectAction() method is called. It in turn calls
>> >>>> sslNegotiation()
>> >>>> >> which in turn sets up the secure socket. In the sslNegotiation()
>> >>>> method, an
>> >>>> >> SSLSocketFactory is not instantiated directly. It is given
by the
>> >>>> >> SSLContext's getSocketFactory() method. This connection
factory
>> >>>> obviously
>> >>>> >> works otherwise the secure connection wouldn't work even
initially.
>> >>>> >>
>> >>>> >> Compare this with what's being done in the execProt() method.
Here
>> the
>> >>>> >> FTPSSocketFactory is instantiated directly with the SSLContext
>> passed
>> >>>> as a
>> >>>> >> constructor argument. Obviously this factory does NOT work...
>> >>>> >>
>> >>>> >> Another question that comes to mind is why the socket factory
>> created
>> >>>> in
>> >>>> >> the sslNegotiation() method is NOT set as the SocketClient's
socket
>> >>>> factory
>> >>>> >> but the one created in the execProt() method is? It turns
out that
>> >>>> >> execProt() calls a generic sendCommand() method that sends
the
>> command
>> >>>> and
>> >>>> >> if the reply is OK, then sets the socket factory to null!
One
>> wonders
>> >>>> why
>> >>>> >> (and there is also a comment in the code: "Check this -
is this
>> >>>> necessary at
>> >>>> >> all?"). Anyway, this is probably the reason why execProt()
needs to
>> >>>> "reset"
>> >>>> >> the connection factory.
>> >>>> >>
>> >>>> >> Still sounds like a commons-net problem though. I'll try
to post
>> this
>> >>>> at
>> >>>> >> their user mailing list.
>> >>>> >>
>> >>>> >> Just one thought. Do we need to reuse the FTPSClient instance?
>> Can't
>> >>>> we
>> >>>> >> just create a fresh new instance everytime  we need to
connect?
>> >>>> >>
>> >>>> >> /Bengt
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> 2010/6/17 Claus Ibsen <claus.ibsen@gmail.com>
>> >>>> >>
>> >>>> >> Hi Bengt
>> >>>> >>>
>> >>>> >>> I think the issue should be created as a ticket for
Apache Commons
>> >>>> >>> Net. Then at least the people there can take a look,
and maybe
>> they
>> >>>> >>> got some ideas. And can fix it in a future release.
>> >>>> >>>
>> >>>> >>> Bengt fell free to experiment yourself with the subclassing,
as it
>> >>>> >>> sounds like a good workaround.
>> >>>> >>>
>> >>>> >>> I assume the FTPSClient doesn't offer other methods
to re connect
>> >>>> >>> which can recover this error?
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> On Wed, Jun 16, 2010 at 2:44 PM, Bengt Rodehav <bengt@rodehav.com
>> >
>> >>>> wrote:
>> >>>> >>> > I have mentioned this problem in a previous conversation
but
>> after
>> >>>> >>> > investigating the subject further I decided to
start a new
>> thread
>> >>>> for
>> >>>> >>> this.
>> >>>> >>> >
>> >>>> >>> > If some kind of problem is encountered while using
ftps (I
>> tested
>> >>>> this
>> >>>> >>> using
>> >>>> >>> > Filezilla server by copying a file that already
existed which
>> >>>> Filezilla
>> >>>> >>> does
>> >>>> >>> > not allow), then at the next attempt to connect
to the ftps
>> server,
>> >>>> I
>> >>>> >>> get
>> >>>> >>> > the following exception in my logfile:
>> >>>> >>> >
>> >>>> >>> > 13:03:27,606 | ERROR | %7Bfile%3Aext%7D |
>> GenericFileOnCompletion
>> >>>> >>>    |
>> >>>> >>> > rg.apache.camel.processor.Logger  248 | Caused
by:
>> >>>> >>> >
>> >>>> [org.apache.camel.component.file.GenericFileOperationFailedException
-
>> >>>> >>> File
>> >>>> >>> > operation failed: null Unconnected sockets not
implemented.
>> Code:
>> >>>> 221]
>> >>>> >>> >
>> >>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>> >>>> >>> File
>> >>>> >>> > operation failed: null Unconnected sockets not
implemented.
>> Code:
>> >>>> 221
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.FtpOperations.connect(FtpOperations.java:108)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT]
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.FtpsOperations.connect(FtpsOperations.java:40)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT]
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.RemoteFileProducer.connectIfNecessary(RemoteFileProducer.java:170)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT]
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.RemoteFileProducer.preWriteCheck(RemoteFileProducer.java:123)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT]
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:75)[65:org.apache.camel.camel-core:2.4.0.SNAPSHOT]
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:49)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT]
>> >>>> >>> > at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:91)[65:org.apache.camel.camel-core:2.4.0.SNAPSHOT]
>> >>>> >>> >
>> >>>> >>> > I added some logging to see the actual stack trace
which is:
>> >>>> >>> >
>> >>>> >>> > java.net.SocketException: Unconnected sockets
not implemented
>> >>>> >>> >        at
>> >>>> javax.net.SocketFactory.createSocket(SocketFactory.java:104)
>> >>>> >>> >        at
>> >>>> >>> >
>> org.apache.commons.net.SocketClient.connect(SocketClient.java:175)
>> >>>> >>> >        at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.FtpOperations.connect(FtpOperations.java:91)
>> >>>> >>> >        at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.FtpsOperations.connect(FtpsOperations.java:40)
>> >>>> >>> >        at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.RemoteFileProducer.connectIfNecessary(RemoteFileProducer.java:170)
>> >>>> >>> >        at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.RemoteFileProducer.preWriteCheck(RemoteFileProducer.java:99)
>> >>>> >>> >        at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:75)
>> >>>> >>> >        at
>> >>>> >>> >
>> >>>> >>>
>> >>>>
>> org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:49)
>> >>>> >>> >
>> >>>> >>> > commons-net SocketClient class tries to call the
socket
>> factory's
>> >>>> >>> >  createSocket() method (with no parameters):
>> >>>> >>> >
>> >>>> >>> >        _socket_= _socketFactory_.createSocket();
>> >>>> >>> >        _socket_.connect(new InetSocketAddress(hostname,
port),
>> >>>> >>> > connectTimeout);
>> >>>> >>> >
>> >>>> >>> > Then intention is to create an "unconnected socket"
(hence the
>> >>>> error
>> >>>> >>> > message) and then subsequently connect it. In
my case, the
>> >>>> connection
>> >>>> >>> > factory being used is an FTPSSocketFactory
>> >>>> >>> > (package org.apache.commons.net.ftp) that inherits
from
>> >>>> >>> > SocketFactory(package javax.net). The SocketFactory
class
>> >>>> implements
>> >>>> >>> the
>> >>>> >>> > createSocket() method but throws the exception
("Unconnected
>> >>>> sockets not
>> >>>> >>> > implemented").
>> >>>> >>> >
>> >>>> >>> > The reason why this happens is that after having
executed the
>> >>>> >>> > FTPSClient.execPROT() method (package
>> org.apache.commons.net.ftp),
>> >>>> with
>> >>>> >>> any
>> >>>> >>> > other parameter than "C" (in my case I used "P"),
the socket
>> >>>> factory is
>> >>>> >>> set
>> >>>> >>> > to an FTPSSocketFactory as follows:
>> >>>> >>> >
>> >>>> >>> >            setSocketFactory(new FTPSSocketFactory(context));
>> >>>> >>> >
>> >>>> >>> > After this, no connection attempt will succeed.
This problem has
>> >>>> been
>> >>>> >>> > introduced recently when Claus (on my initiative)
added support
>> for
>> >>>> >>> secure
>> >>>> >>> > data channel in ftps. I'm thus to blame...
>> >>>> >>> >
>> >>>> >>> > I'm not sure how this should be fixed.
>> >>>> >>> >
>> >>>> >>> > It's a bit strange that all the connect methods
>> >>>> >>> > in org.apache.commons.net.SocketClient (which
is the ultimate
>> base
>> >>>> class
>> >>>> >>> of
>> >>>> >>> > FTPSClient) always try to create an unconnected
socket first and
>> >>>> then
>> >>>> >>> > connects it. If commons-net uses this pattern,
then it should
>> make
>> >>>> sure
>> >>>> >>> that
>> >>>> >>> > all its connection factor's support unconnected
sockets but the
>> >>>> >>> > FTPSSocketFactory dosn't. This sounds like a bug
in commons-net.
>> >>>> >>> >
>> >>>> >>> > Knowing that commons-net doesn't release very
frequently I think
>> we
>> >>>> need
>> >>>> >>> a
>> >>>> >>> > workaround for this. I guess it would be possible
to subclass
>> the
>> >>>> >>> FTPClient
>> >>>> >>> > class and override all the connect methods to
make sure that no
>> >>>> attempt
>> >>>> >>> is
>> >>>> >>> > made to create unconnected sockets. We would then
change the
>> >>>> >>> > createFtpClient() method in the FtpsEndpoint class
to
>> instantiate
>> >>>> our
>> >>>> >>> own
>> >>>> >>> > subclass instead of an FTPSClient. Doesn't sound
like a nice
>> clean
>> >>>> >>> solution
>> >>>> >>> > though.
>> >>>> >>> >
>> >>>> >>> > Any ideas?
>> >>>> >>> >
>> >>>> >>> > /Bengt
>> >>>> >>> >
>> >>>> >>>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> --
>> >>>> >>> Claus Ibsen
>> >>>> >>> Apache Camel Committer
>> >>>> >>>
>> >>>> >>> Author of Camel in Action: http://www.manning.com/ibsen/
>> >>>> >>> Open Source Integration: http://fusesource.com
>> >>>> >>> Blog: http://davsclaus.blogspot.com/
>> >>>> >>> Twitter: http://twitter.com/davsclaus
>> >>>> >>>
>> >>>> >>
>> >>>> >>
>> >>>> >
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Claus Ibsen
>> >>>> Apache Camel Committer
>> >>>>
>> >>>> Author of Camel in Action: http://www.manning.com/ibsen/
>> >>>> Open Source Integration: http://fusesource.com
>> >>>> Blog: http://davsclaus.blogspot.com/
>> >>>> Twitter: http://twitter.com/davsclaus
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message