camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claus Ibsen (JIRA)" <>
Subject [jira] [Resolved] (CAMEL-9478) GenericFileProcessStrategySupport renameFile is destructive when the source file is missing
Date Tue, 09 Feb 2016 09:40:18 GMT


Claus Ibsen resolved CAMEL-9478.
    Resolution: Won't Fix

We should not hide that fact. When using windows then files may be locked due some resource
is not properly closed.

> GenericFileProcessStrategySupport renameFile is destructive when the source file is missing
> -------------------------------------------------------------------------------------------
>                 Key: CAMEL-9478
>                 URL:
>             Project: Camel
>          Issue Type: Bug
>    Affects Versions: 2.16.0
>         Environment: Windows, OSX, JVM 1.8.0_20
>            Reporter: Philippe Khalife
>              Labels: file, idempotent, threading
> Given this setup:
> String fileConsumerURI="file://spool?move=.done&doneFileName=${file:name}.done&sortBy=file:modified&maxMessagesPerPoll=5000&eagerMaxMessagesPerPoll=true"
> RouteDefinition route = from(fileConsumerURI).routeId("MyRouteId").to("metrics:counter:MyRoute.counter?increment=1");
> route.idempotentConsumer(header("CamelFileName"))
>                                         .messageIdRepository(memoryIdempotentRepository)
>                                         .threads(CONFIGURABLE_NUMBER)
>                                         .unmarshal()
>                                         .serialization()
>                                         .process(MY_SIMPLE_PROCESSOR)
>                                         .recipientList(header("HEADER_CREATED_BY_MY_SIMPLE_PROCESSOR"));
> Can occasionally lead to a destructive action by "GenericFileProcessStrategySupport.renameFile"
when it attempts to delete the "to.getAbsoluteFilePath()" while the source file "from.getAbsoluteFilePath()"
does not even exist.
> I think is a bug on its own regardless of how we got there. The GenericFileProcessStrategySupport.renameFile
method should check that from.getAbsoluteFilePath() exists BEFORE it attempts the delete.

> How did we get there? - I need to do multi-threaded processing of stored files in a spool,
the files  contain meta data about their intended destination, AND using a configurable number
of threads to do the work.
> Why the idempotentConsumer in the route? I needed to control the size of the memoryIdempotentRepository,
which is NOT configurable in the file component.
> In this setup the file component seems to have a tendency to occasionally send a file
to processing more than once, the idempotentConsumer takes care of that pretty well, but NOT
when I choose to move the files instead of delete them as per the route above.
> This can be reproduced on its own by running a lot of data and having varied delays in
the processing. After much waiting and catching the real bug, I came up with a simulation
for it.
> By debugging and suspending I can simulate when the problem happens. This setup shows
the consequences of files being pushed in exchanges more than once by the file consumer.
> Let file F1 get processed, and moved, copy it back to the spool manually (this part is
just to force the file being listed for processing more than once), now the idempotentConsumer
will reject it, while suspeneded manually move/delete F1 file from the spool, now the file
consumer's GenericFileProcessStrategySupport.renameFile will end up eagerly deleting an already
processed / moved files while the F1 file does not exist.  
> I did on a temporary basis use the file consumer's idempotent setting to avoid this issue
as a workaround, but this is not ideal for my workflow.
> I have a number of recommendations:
> 1) Fix this bug: The GenericFileProcessStrategySupport.renameFile method should check
that from.getAbsoluteFilePath() exists BEFORE it attempts the delete. 
> 2) Have a configurable idempotent repository for the file consumer.
> A sample stack trace:
> WARN  GenericFileOnCompletion:135 - Error during commit. Exchange[ID-I-local-53638-1450384044069-0-170][ID-I-local-53638-1450384044069-0-5].
Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot rename
file: GenericFile[ID-I-local-53638-1450384044069-0-5] to: GenericFile[.sent/ID-I-local-53638-1450384044069-0-5]]
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> rename file: GenericFile[ID-I-local-53638-1450384044069-0-5] to:
> GenericFile[.sent/ID-I-local-53638-1450384044069-0-5]
>         at
> org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(
>         at
> org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(
>         at
> org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(
>         at
> org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(
>         at
> org.apache.camel.component.file.GenericFileOnCompletion.onComplete(
>         at
> org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(
>         at
> org.apache.camel.impl.DefaultUnitOfWork.done(
>         at
> org.apache.camel.util.UnitOfWorkHelper.doneUow(
>         at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(
>         at
> org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(
>         at
> org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(
>         at org.apache.camel.processor.Pipeline.process(
>         at
> org.apache.camel.processor.CamelInternalProcessor.process(
>         at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(
>         at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(
>         at
> org.apache.camel.component.file.GenericFileConsumer.poll(
>         at
> org.apache.camel.impl.ScheduledPollConsumer.doRun(
>         at
>         at
> java.util.concurrent.Executors$
>         at java.util.concurrent.FutureTask.runAndReset(
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
>         at
> java.util.concurrent.ThreadPoolExecutor$
>         at

This message was sent by Atlassian JIRA

View raw message