camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philippe Khalife (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CAMEL-9478) GenericFileProcessStrategySupport renameFile is destructive when the source file is missing
Date Mon, 04 Jan 2016 20:20:40 GMT
Philippe Khalife created CAMEL-9478:
---------------------------------------

             Summary: GenericFileProcessStrategySupport renameFile is destructive when the
source file is missing
                 Key: CAMEL-9478
                 URL: https://issues.apache.org/jira/browse/CAMEL-9478
             Project: Camel
          Issue Type: Bug
    Affects Versions: 2.16.0
         Environment: Windows, OSX
            Reporter: Philippe Khalife



Given this setup:

String fileConsumerURI="file://spool?move=.done&doneFileName=${file:name}.done&sortBy=file:modified&maxMessagesPerPoll=5000&eagerMaxMessagesPerPoll=true"

RouteDefinition route = from(fileConsumerURI).routeId("MyRouteId").to("metrics:counter:MyRoute.counter?increment=1");

route.idempotentConsumer(header("CamelFileName"))
                                        .messageIdRepository(memoryIdempotentRepository)
                                        .threads(CONFIGURABLE_NUMBER)
                                        .unmarshal()
                                        .serialization()
                                        .process(MY_SIMPLE_PROCESSOR)
                                        .recipientList(header("HEADER_CREATED_BY_MY_SIMPLE_PROCESSOR"));


Can occasionally lead to a destructive action by "GenericFileProcessStrategySupport.renameFile"
when it attempts to delete the "to.getAbsoluteFilePath()" while the source file "from.getAbsoluteFilePath()"
does not even exist.
I think is a bug on its own regardless of how we got there. The GenericFileProcessStrategySupport.renameFile
method should check that from.getAbsoluteFilePath() exists BEFORE it attempts the delete.


How did we get there? - I need to do multi-threaded processing of stored files in a spool,
the files  contain meta data about their intended destination, AND using a configurable number
of threads to do the work.
Why the idempotentConsumer in the route? I needed to control the size of the memoryIdempotentRepository,
which is NOT configurable in the file component.

In this setup the file component seems to have a tendency to occasionally send a file to processing
more than once, the idempotentConsumer takes care of that pretty well, but NOT when I choose
to move the files instead of delete them as per the route above.

This can be reproduced on its own by running a lot of data and having varied delays in the
processing. After much waiting and catching the real bug, I came up with a simulation for
it.

By debugging and suspending I can simulate when the problem happens. This setup shows the
consequences of files being pushed in exchanges more than once by the file consumer.

Let file F1 get processed, and moved, copy it back to the spool manually (this part is just
to force the file being listed for processing more than once), now the idempotentConsumer
will reject it, while suspeneded manually move/delete F1 file from the spool, now the file
consumer's GenericFileProcessStrategySupport.renameFile will end up eagerly deleting an already
processed / moved files while the F1 file does not exist.  

I did on a temporary basis use the file consumer's idempotent setting to avoid this issue
as a workaround, but this is not ideal for my workflow.

I have a number of recommendations:
1) Fix this bug: The GenericFileProcessStrategySupport.renameFile method should check that
from.getAbsoluteFilePath() exists BEFORE it attempts the delete. 
2) Have a configurable idempotent repository for the file consumer.


A sample stack trace:

WARN  GenericFileOnCompletion:135 - Error during commit. Exchange[ID-I-local-53638-1450384044069-0-170][ID-I-local-53638-1450384044069-0-5].
Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename
file: GenericFile[ID-I-local-53638-1450384044069-0-5] to: GenericFile[.sent/ID-I-local-53638-1450384044069-0-5]]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
rename file: GenericFile[ID-I-local-53638-1450384044069-0-5] to:
GenericFile[.sent/ID-I-local-53638-1450384044069-0-5]
        at
org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:115)
        at
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
        at
org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
        at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
        at
org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
        at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
        at
org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
        at
org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
        at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:653)
        at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:608)
        at
org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:239)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:109)
        at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
        at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442)
        at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214)
        at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178)
        at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
        at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message