camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sagy <sagyroz...@gmail.com>
Subject Re: File consumer with noop=true&recursive=true
Date Tue, 13 Apr 2010 15:53:49 GMT

Here is a small junit test which explains the problem:

public void testIdempotentConsumerDoesntIgnoreRecursiveDuplicates() throws
Exception {
		String newFile1 = "some_dir/some_file3";
		String newFile2 = "some_file3";
		mockRecognizerService.setExpectedMessageCount(2);
		dropFileInTargetDir(newFile1);
		dropFileInTargetDir(newFile2);
		Thread.sleep(4000);
		mockRecognizerService.assertIsSatisfied();
	}

dropFileInTargetDir - creates a file under the directory which the file
component is listening to.
This test fails.



sagy wrote:
> 
> exactly. Lets assume a directory root called /test
> the writing application will write the following files:
> /test/file1.txt 
> /test/file2.txt 
> /test/sub-folder/file1.txt 
> 
> etc..
> My application will get an event for 
> /test/file1.txt 
> /test/file2.txt 
> 
> but not for 
> /test/sub-folder/file1.txt 
> 
> since the IdempotentRepository already contains an entry called
> "file1.txt". It was created for the /test/file1.txt event.
> 
> 
> 
> 
> 
> Claus Ibsen-2 wrote:
>> 
>> On Tue, Apr 13, 2010 at 5:10 PM, sagy <sagyrozman@gmail.com> wrote:
>>>
>>> I'm trying to consume files from it not skip it.
>>> This is a general description of the situation:
>>>
>>> There is another application with write permissions which create files
>>> in
>>> that directory and sub directories.
>>> My application only has read permissions but I still need to consume the
>>> files from that directory structure.
>>>
>>> That's why I'm using the "noop=true" flag. I can't move the files or
>>> delete
>>> them.
>>> It seems from the documentation that noop is my only choice.
>>> But a combination of "recursive=true" and "noop=true" causes the problem
>>> I've described in the original post.
>>>
>> 
>> And that app can write files with the same name?
>> 
>> 
>> 
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Tue, Apr 13, 2010 at 3:33 PM, sagy <sagyrozman@gmail.com> wrote:
>>>>>
>>>>> I'm actually trying to consume files from a directory recursively and
>>>>> the
>>>>> directory has read-only permissions.
>>>>> Is there another way to do it without using noop=true&recursive=true
?
>>>>>
>>>>
>>>> You can filter out the read-only directory if you should "skip" it.
>>>>
>>>>
>>>>> Thanks,
>>>>> Sagy
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Yeah its on purpose for some reason, which as I can't really recall
>>>>>> right
>>>>>> now.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 13, 2010 at 2:33 PM, sagy <sagyrozman@gmail.com>
wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> I'm using Camel 2.2 and have the following route:
>>>>>>>
>>>>>>> from("file://test?recursive=true&noop=true").process(...
>>>>>>>
>>>>>>> When I drop a file f1.txt into the test folder the processor
gets
>>>>>>> called.
>>>>>>> When I drop the file f2.txt into a sub folder of test folder
for
>>>>>>> example
>>>>>>> test/sub/f2.txt ,again the processor gets called.
>>>>>>> But when I drop the file f1.txt into test/sub the processor doesn't
>>>>>>> get
>>>>>>> called.
>>>>>>> This is happening because of line 115 in camel-core
>>>>>>> org.apache.camel.component.file.GenericFileOnCompletion:
>>>>>>>
>>>>>>>            // only add to idempotent repository if we could
process
>>>>>>> the
>>>>>>> file
>>>>>>>            // only use the filename as the key as the file
could be
>>>>>>> moved
>>>>>>> into a done folder
>>>>>>>          
>>>>>>>  endpoint.getIdempotentRepository().add(file.getFileName());
>>>>>>>
>>>>>>> The repository stores just the file name without the path.
>>>>>>> Therefore if 2 files with the same name are dropped to 2 different
>>>>>>> sub
>>>>>>> directories only the first one will be processed.
>>>>>>> Is this the intended behavior?
>>>>>>> Is there a way to work around it?
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28229501.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28230077.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28231610.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>> 
>> 
>> 
>> -- 
>> Claus Ibsen
>> Apache Camel Committer
>> 
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/File-consumer-with-noop%3Dtrue-recursive%3Dtrue-tp28229501p28232225.html
Sent from the Camel - Users mailing list archive at Nabble.com.


Mime
View raw message