camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: SFTP rename problems
Date Wed, 21 Apr 2010 06:10:04 GMT
Hi

I tacked down an issue in the camel-core
https://issues.apache.org/activemq/browse/CAMEL-2662

Can you try to test with latest code on your system?



On Wed, Apr 21, 2010 at 6:13 AM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
> On Wed, Apr 21, 2010 at 3:31 AM, watcher <patrick@daly.ws> wrote:
>>
>> I tried upgrading to jcraft v0.1.42 but the problem still exists.
>>
>> I can recreate the problem by copying a large file > 1MB onto the Sftp
>> server, and keeping the file open until after the next poll.
>>
>
> Ah makes sense. The sftp component is trying to get exclusive
> permission to the file.
> It does this by trying to rename the file to *.camelExclusiveReadLock.
> And the SFTP library is denying this because
> the file is actually opened by another program.
>
> What Camel should have do is to silently ignore this error instead of
> throwing it as an ERROR in the log.
> I will look into this.
>
> But if you stop having the file opened then the SFTP component should
> be able to rename the file on the next pool.
> Or a bit thereafter until the Windows file system allows to rename the
> file again.
>
>
>
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> Btw you can try upgrading to a new jcraft .jar. v0.1.42 is out and we
>>> have upgraded to that in Camel 2.3.
>>> Camel 2.2 was using 0.1.40.
>>>
>>>
>>> On Thu, Apr 15, 2010 at 4:21 PM, watcher <patrick@daly.ws> wrote:
>>>>
>>>> Hi Claus
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> The status code 3 is SSH_ERROR_PERMISSION_DENIED (the authenticated user
>>>> does not have sufficient permissions to perform the operation.)
>>>>
>>>> http://www.eldos.com/documentation/sbb/documentation/ref_err_sftperrorcodes.html
>>>> http://www.eldos.com/documentation/sbb/documentation/ref_err_sftperrorcodes.html
>>>>
>>>> Yes the problem happens for all other incoming files, restarting the
>>>> application seems to repair the problem.
>>>> I dont see in the logs any sign of camel disconnecting from the session
>>>> and
>>>> retrying to login.
>>>>
>>>> Most of our clients write to the incoming sftp folder using a temporary
>>>> filename before renaming it however that might not be possible to for one
>>>> of
>>>> our clients in the short to medium term.
>>>>
>>>> I didnt think it was a issue as the readlock has work successfully with
>>>> the
>>>> file component and I aware they share a common code base.
>>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> The SFTP library is returning an exception with status code 3. You may
>>>>> dig around what that means.
>>>>>
>>>>> When this problem occurs, does the same problem occur on next poll?
>>>>> The ftp component have build in self healing, but that is limited to
>>>>> the ftp connection/session.
>>>>> So in case there is something wrong with the session it re login
>>>>> automatic.
>>>>>
>>>>> Also you can use readLock=none as uri parameter to not use locks at
>>>>> all. You can do this if you do not have a race condition
>>>>> where another party is writing a new file, which you then want to
>>>>> download.
>>>>>
>>>>> Its actually best to let the other party write to another directory
>>>>> and then when the file is complete, move it in place.
>>>>> Or write using a temporary filename, and then rename when write is
>>>>> complete.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Apr 15, 2010 at 1:37 PM, watcher <patrick@daly.ws> wrote:
>>>>>>
>>>>>> Apache Camel 2.2
>>>>>> Windows Server 2003
>>>>>>
>>>>>> I'm running an application that polls an SFTP server. I works to
a
>>>>>> point
>>>>>> but
>>>>>> after some time the applications fails trying to get an exclusive
lock
>>>>>> on
>>>>>> the file. Whick seems to indicate that it is a permission problem.
>>>>>> However
>>>>>> restarting the application (camel) fixes the issue. Could it be a
fault
>>>>>> with
>>>>>> camel?
>>>>>>
>>>>>> [code]
>>>>>>
>>>>>> <camel:camelContext xmlns="http://camel.apache.org/schema/spring">
>>>>>>  <camel:routeBuilder ref="myRouter"/>
>>>>>>        <camel:endpoint id="printerIncoming"
>>>>>> uri="sftp://${printerIncoming}@${ipaddress}?password=password&amp;binary=true&amp;recursive=true&amp;delay=10000&amp;initialDelay=2000&amp;move=.done&amp;knownHostsFile=c://known_host&amp;filter=#myAntFilter"/>
>>>>>> </camel:camelContext>
>>>>>>
>>>>>>
>>>>>>        <!-- we use the AntPathMatcherRemoteFileFilter to use
ant paths
>>>>>> for
>>>>>> includes and exlucde -->
>>>>>>        <bean id="myAntFilter"
>>>>>> class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
>>>>>>            <property name="includes"
>>>>>> value="**/incoming/*.PDF,**/incoming/*.pdf"/>
>>>>>>            <property name="excludes" value="**/outgoing/**"/>
>>>>>>
>>>>>>        </bean>
>>>>>> [/code]
>>>>>>
>>>>>> [code]
>>>>>> 15 Apr 2010 10:17:57,077 ERROR [Camel thread 0: SftpComponent]
>>>>>> SftpConsumer.log(248) | Caused by:
>>>>>> [org.apache.camel.component.file.GenericFileOperationFailedException
-
>>>>>> Cannot rename file from: user/incoming/test_file.pdf to:
>>>>>> test_file.pdf.camelExclusiveReadLock]
>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>> Cannot
>>>>>> rename file from: user/incoming/test_file.pdf to:
>>>>>> test_file.pdf.camelExclusiveReadLock
>>>>>>        at
>>>>>> org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:205)
>>>>>>        at
>>>>>> org.apache.camel.component.file.strategy.GenericFileRenameExclusiveReadLockStrategy.acquireExclusiveReadLock(GenericFileRenameExclusiveReadLockStrategy.java:68)
>>>>>>        at
>>>>>> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.begin(GenericFileProcessStrategySupport.java:44)
>>>>>>        at
>>>>>> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.begin(GenericFileRenameProcessStrategy.java:38)
>>>>>>        at
>>>>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:221)
>>>>>>        at
>>>>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:137)
>>>>>>        at
>>>>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:103)
>>>>>>        at
>>>>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106)
>>>>>>        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>>> Source)
>>>>>>        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown
>>>>>> Source)
>>>>>>        at java.util.concurrent.FutureTask.runAndReset(Unknown
Source)
>>>>>>        at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>>>>> Source)
>>>>>>        at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>>>>>> Source)
>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>> Source)
>>>>>>        at java.lang.Thread.run(Unknown Source)
>>>>>> Caused by: 3:
>>>>>>        at
>>>>>> com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2287)
>>>>>>        at com.jcraft.jsch.ChannelSftp.rename(ChannelSftp.java:1479)
>>>>>>        at
>>>>>> org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:202)
>>>>>>        ... 16 more
>>>>>> [/code]
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/SFTP-rename-problems-tp28254146p28254146.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/SFTP-rename-problems-tp28254146p28255938.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context: http://old.nabble.com/SFTP-rename-problems-tp28254146p28287939.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message