camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sven Nold <Sven.N...@isb-ag.de>
Subject AW: camel 2.13.2 zipdataformat
Date Tue, 02 Sep 2014 13:13:47 GMT
Hi, 

same result, but workaround is to use zipSplitter like this:

<bean id="zipSplitter" class="org.apache.camel.dataformat.zipfile.ZipSplitter" />

        <route id="readZip" streamCache="true">
            <from uri="file://target?fileName=any.xxx&amp;moveFailed=.error&amp;move=.done&amp;delay=5000"/>
            <!-- <unmarshal ref="zipFileDataFormat"/> -->
            <camel:split streaming="true">
                <!-- <camel:simple>${body}</camel:simple> -->
                <camel:ref>zipSplitter</camel:ref>
                <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
                <camel:to uri="file://target/unzipped"/> 
            </camel:split> 
            <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
        </route>

ZipSplitter seems not to have this issue. Also works for 2.13.2. which could be helpful for
you @Thomas.

So from my point of view CAMEL-7577 is not RESOLVED.

Best regards,

-----Ursprüngliche Nachricht-----
Von: Claus Ibsen [mailto:claus.ibsen@gmail.com] 
Gesendet: Dienstag, 2. September 2014 14:29
An: users@camel.apache.org
Betreff: Re: camel 2.13.2 zipdataformat

You need to build the code to make sure the snapshot is using recent code.

On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sven.Nold@isb-ag.de> wrote:
> Hi Claus,
>
> I changed to 2.13.3-SNAPSHOT but still the same issue.
>
> Best regards,
>
> Sven
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Gesendet: Dienstag, 2. September 2014 13:52
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> Hi
>
> See CAMEL-7577
>
> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sven.Nold@isb-ag.de> wrote:
>> Hi,
>>
>> interesting is also that if I only have on zipEntry and define the zipDataFormat
as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on
the file:
>>
>>     <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>         <property name="usingIterator" value="false"/>
>>     </bean>
>>
>>     <bean id="zipAggregationStrategy"
>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStr
>> a
>> tegy"/>
>>
>>     <camelContext xmlns="http://camel.apache.org/schema/spring">
>>         <!-- write the zip so we can process something repeatedly -->
>>         <route id="writeZip">
>>             <from uri="file://src/main/data?antInclude=message1.xml&amp;noop=true"/>
>>             <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true"
eagerCheckCompletion="true" completionSize="2">
>>                 <correlationExpression>
>>                         <constant>true</constant>
>>                 </correlationExpression>
>>                 <to uri="file://target?fileName=any.xxx"/>
>>             </aggregate>
>>         </route>
>>
>>         <!-- extract the zip -->
>>         <route id="readZip" streamCache="true">
>>             <from uri="file://target?fileName=any.xxx&amp;moveFailed=.error&amp;move=.done&amp;delay=5000"/>
>>             <unmarshal ref="zipFileDataFormat"/>
>>             <!--<camel:split streaming="true">
>>                 <camel:simple>${body}</camel:simple>
>>                 <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
>>                 <camel:to uri="file://target/unzipped"/>
>>             </camel:split> -->
>>                         <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
>>         </route>
>>     </camelContext>
>>
>> Works fine.
>>
>> Best regards,
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>> Gesendet: Dienstag, 2. September 2014 13:38
>> An: users@camel.apache.org
>> Betreff: AW: camel 2.13.2 zipdataformat
>>
>> Hi,
>>
>> thanks for this test now I think here is the problem:
>>
>> [l-1) thread #1 - file://target] GenericFileOnCompletion        WARN  Error during
commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException
- Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip
to target\.done\any.zip]
>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming
file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>>         at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>>         at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>>         at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>         at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>>         at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>>         at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>>         at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>>         at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>>         at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>>         at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>>         at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>>         at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>>         at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>>         at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>>         at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>>         at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>>         at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>>         at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip'
to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip'
after copy succeeded
>>         at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>>         at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>>         at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>>         ... 25 more
>>
>> This is why it grabs the file all the time.
>>
>> Regarding your questions:
>>
>> 1) well thanks to your test you can see error above
>> 2) yes it is processing it over and over again
>> 3) no doesn't move to .error
>> 4) no file is not updated, in your test case I am not quite sure, will increase delay
or as you suggested preMove option.
>>
>> Due to known problems off windows and zip files I even renamed it to .xxx to keep
away explorer from analyzing, but still it seems there is a handle on that file.
>> And again if I remove the unmarshal and split it works fine.
>>
>> Thanks for any help.
>>
>> Best regards,
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 10:37
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> Hi,
>>
>> I just added an example at
>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>> This example creates the zip file first (so that it's repeatable) & then does
pretty much what you're doing: read from file uri, unmarshal, log each file contents (careful
with large files) & finally writes each file to a file. Seems to be doing what it should...
>>
>> Few questions if the example doesn't help you:
>>
>>    - Are there any hints in the logs (try increasing logging level if not)?
>>    - If the file is back in the source dir does it try to process again?
>>    - Is there anything in the .error (moveFailed dir)?
>>    - Could the source file be being updated while you're processing it - if
>>    so, can you try the preMove option?
>>
>> Thanks,
>>
>>
>> On 2 September 2014 08:46, Walzer, Thomas 
>> <thomas.walzer@integratix.net>
>> wrote:
>>
>>> Hi Sven,
>>>
>>> I seem to have a similar problem. I also use 2.13.2.
>>> My code is almost identical. The only difference is my in-url, that 
>>> looks
>>> like:
>>>
>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${f
>>> i l e:name}.${date:now:HHmmss}&amp;antInclude=*.ZIP
>>>
>>> I get a random number of retrievals (about 4-20) and therefore the 
>>> same number of files in the „archive“ directory. The timestamps are 
>>> 6 seconds apart.
>>>
>>> Cheers, Thomas.
>>>
>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sven.Nold@isb-ag.de>:
>>>
>>> > Hi Guys,
>>> >
>>> > I am trying to read Zipentries and process them separately. But 
>>> > using
>>> the file component it seems the file is not correctly closed. I can 
>>> see the file for a few seconds in the .done and source directory. 
>>> But then it disappears in the .done folder and is back in the source directory
:
>>> >
>>> >
>>> > <bean id="zipFileDataFormat"
>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>> >    <property name="usingIterator" value="true" />  </bean>
>>> >
>>> > <route id="readZip">
>>> >      <from
>>> uri="file://C:/temp/?fileName=any.zip&amp;moveFailed=.error&amp;move=.done&amp;delay=10000"
>>> />
>>> >      <unmarshal ref="zipFileDataFormat" />
>>> > <!--       <split streaming="true"> -->
>>> > <!--             <simple>$simple{body}</simple> -->
>>> > <!--             <to
>>> uri="log://OUT?level=INFO&amp;showAll=true&amp;multiline=true" />

>>> -->
>>> > <!--       </split> -->
>>> >      <to uri="log://OUT?level=INFO&amp;showAll=true&amp;multiline=true"
>>> />
>>> > </route>
>>> >
>>> > If I remove the unmarshal the file component works fine (it moves
>>> processed file to .done).
>>> >
>>> > Any Ideas? I guess I could do a workaround by using noop=true and 
>>> > moving
>>> the file myself, but it would be nice if it works as expected.
>>> >
>>> > Best regards,
>>> >
>>> > Sven
>>>
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Mime
View raw message