camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From watcher <patr...@daly.ws>
Subject Re: SFTP rename problems
Date Tue, 27 Apr 2010 10:35:20 GMT

Issue Fixed.



Claus Ibsen-2 wrote:
> 
> Hi
> 
> I tacked down an issue in the camel-core
> https://issues.apache.org/activemq/browse/CAMEL-2662
> 
> Can you try to test with latest code on your system?
> 
> 
> 
> On Wed, Apr 21, 2010 at 6:13 AM, Claus Ibsen <claus.ibsen@gmail.com>
> wrote:
>> On Wed, Apr 21, 2010 at 3:31 AM, watcher <patrick@daly.ws> wrote:
>>>
>>> I tried upgrading to jcraft v0.1.42 but the problem still exists.
>>>
>>> I can recreate the problem by copying a large file > 1MB onto the Sftp
>>> server, and keeping the file open until after the next poll.
>>>
>>
>> Ah makes sense. The sftp component is trying to get exclusive
>> permission to the file.
>> It does this by trying to rename the file to *.camelExclusiveReadLock.
>> And the SFTP library is denying this because
>> the file is actually opened by another program.
>>
>> What Camel should have do is to silently ignore this error instead of
>> throwing it as an ERROR in the log.
>> I will look into this.
>>
>> But if you stop having the file opened then the SFTP component should
>> be able to rename the file on the next pool.
>> Or a bit thereafter until the Windows file system allows to rename the
>> file again.
>>
>>
>>
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> Btw you can try upgrading to a new jcraft .jar. v0.1.42 is out and we
>>>> have upgraded to that in Camel 2.3.
>>>> Camel 2.2 was using 0.1.40.
>>>>
>>>>
>>>> On Thu, Apr 15, 2010 at 4:21 PM, watcher <patrick@daly.ws> wrote:
>>>>>
>>>>> Hi Claus
>>>>>
>>>>> Thanks for the quick reply.
>>>>>
>>>>> The status code 3 is SSH_ERROR_PERMISSION_DENIED (the authenticated
>>>>> user
>>>>> does not have sufficient permissions to perform the operation.)
>>>>>
>>>>> http://www.eldos.com/documentation/sbb/documentation/ref_err_sftperrorcodes.html
>>>>> http://www.eldos.com/documentation/sbb/documentation/ref_err_sftperrorcodes.html
>>>>>
>>>>> Yes the problem happens for all other incoming files, restarting the
>>>>> application seems to repair the problem.
>>>>> I dont see in the logs any sign of camel disconnecting from the
>>>>> session
>>>>> and
>>>>> retrying to login.
>>>>>
>>>>> Most of our clients write to the incoming sftp folder using a
>>>>> temporary
>>>>> filename before renaming it however that might not be possible to for
>>>>> one
>>>>> of
>>>>> our clients in the short to medium term.
>>>>>
>>>>> I didnt think it was a issue as the readlock has work successfully
>>>>> with
>>>>> the
>>>>> file component and I aware they share a common code base.
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> The SFTP library is returning an exception with status code 3. You
>>>>>> may
>>>>>> dig around what that means.
>>>>>>
>>>>>> When this problem occurs, does the same problem occur on next poll?
>>>>>> The ftp component have build in self healing, but that is limited
to
>>>>>> the ftp connection/session.
>>>>>> So in case there is something wrong with the session it re login
>>>>>> automatic.
>>>>>>
>>>>>> Also you can use readLock=none as uri parameter to not use locks
at
>>>>>> all. You can do this if you do not have a race condition
>>>>>> where another party is writing a new file, which you then want to
>>>>>> download.
>>>>>>
>>>>>> Its actually best to let the other party write to another directory
>>>>>> and then when the file is complete, move it in place.
>>>>>> Or write using a temporary filename, and then rename when write is
>>>>>> complete.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 15, 2010 at 1:37 PM, watcher <patrick@daly.ws>
wrote:
>>>>>>>
>>>>>>> Apache Camel 2.2
>>>>>>> Windows Server 2003
>>>>>>>
>>>>>>> I'm running an application that polls an SFTP server. I works
to a
>>>>>>> point
>>>>>>> but
>>>>>>> after some time the applications fails trying to get an exclusive
>>>>>>> lock
>>>>>>> on
>>>>>>> the file. Whick seems to indicate that it is a permission problem.
>>>>>>> However
>>>>>>> restarting the application (camel) fixes the issue. Could it
be a
>>>>>>> fault
>>>>>>> with
>>>>>>> camel?
>>>>>>>
>>>>>>> [code]
>>>>>>>
>>>>>>> <camel:camelContext xmlns="http://camel.apache.org/schema/spring">
>>>>>>>  <camel:routeBuilder ref="myRouter"/>
>>>>>>>        <camel:endpoint id="printerIncoming"
>>>>>>> uri="sftp://${printerIncoming}@${ipaddress}?password=password&amp;binary=true&amp;recursive=true&amp;delay=10000&amp;initialDelay=2000&amp;move=.done&amp;knownHostsFile=c://known_host&amp;filter=#myAntFilter"/>
>>>>>>> </camel:camelContext>
>>>>>>>
>>>>>>>
>>>>>>>        <!-- we use the AntPathMatcherRemoteFileFilter
to use ant
>>>>>>> paths
>>>>>>> for
>>>>>>> includes and exlucde -->
>>>>>>>        <bean id="myAntFilter"
>>>>>>> class="org.apache.camel.component.file.AntPathMatcherGenericFileFilter">
>>>>>>>            <property name="includes"
>>>>>>> value="**/incoming/*.PDF,**/incoming/*.pdf"/>
>>>>>>>            <property name="excludes" value="**/outgoing/**"/>
>>>>>>>
>>>>>>>        </bean>
>>>>>>> [/code]
>>>>>>>
>>>>>>> [code]
>>>>>>> 15 Apr 2010 10:17:57,077 ERROR [Camel thread 0: SftpComponent]
>>>>>>> SftpConsumer.log(248) | Caused by:
>>>>>>> [org.apache.camel.component.file.GenericFileOperationFailedException
>>>>>>> -
>>>>>>> Cannot rename file from: user/incoming/test_file.pdf to:
>>>>>>> test_file.pdf.camelExclusiveReadLock]
>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot
>>>>>>> rename file from: user/incoming/test_file.pdf to:
>>>>>>> test_file.pdf.camelExclusiveReadLock
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:205)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.strategy.GenericFileRenameExclusiveReadLockStrategy.acquireExclusiveReadLock(GenericFileRenameExclusiveReadLockStrategy.java:68)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.begin(GenericFileProcessStrategySupport.java:44)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.begin(GenericFileRenameProcessStrategy.java:38)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:221)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:137)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:103)
>>>>>>>        at
>>>>>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106)
>>>>>>>        at
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>>>>>>> Source)
>>>>>>>        at
>>>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown
>>>>>>> Source)
>>>>>>>        at java.util.concurrent.FutureTask.runAndReset(Unknown
>>>>>>> Source)
>>>>>>>        at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown
>>>>>>> Source)
>>>>>>>        at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown
>>>>>>> Source)
>>>>>>>        at
>>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>>>>>>> Source)
>>>>>>>        at
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
>>>>>>> Source)
>>>>>>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>>>>>>> Source)
>>>>>>>        at java.lang.Thread.run(Unknown Source)
>>>>>>> Caused by: 3:
>>>>>>>        at
>>>>>>> com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2287)
>>>>>>>        at com.jcraft.jsch.ChannelSftp.rename(ChannelSftp.java:1479)
>>>>>>>        at
>>>>>>> org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:202)
>>>>>>>        ... 16 more
>>>>>>> [/code]
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/SFTP-rename-problems-tp28254146p28254146.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/SFTP-rename-problems-tp28254146p28255938.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/SFTP-rename-problems-tp28254146p28287939.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
> 
> 
> 
> -- 
> Claus Ibsen
> Apache Camel Committer
> 
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
> 
> 

-- 
View this message in context: http://old.nabble.com/SFTP-rename-problems-tp28254146p28375361.html
Sent from the Camel - Users mailing list archive at Nabble.com.


Mime
View raw message