camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: camel 2.13.2 zipdataformat
Date Tue, 02 Sep 2014 14:07:17 GMT
Hi

Thanks. I suspect its because this code creates a new input stream in
the message body
https://github.com/apache/camel/blob/master/components/camel-zipfile/src/main/java/org/apache/camel/dataformat/zipfile/ZipIterator.java#L106

That is not explicit closed.

Can you try changing

                    answer.setBody(new ZipInputStreamWrapper(zipInputStream));

To

                    answer.setBody(zipInputStream);

And give it a test on your Windows system?

On Tue, Sep 2, 2014 at 3:13 PM, Sven Nold <Sven.Nold@isb-ag.de> wrote:
> Hi,
>
> same result, but workaround is to use zipSplitter like this:
>
> <bean id="zipSplitter" class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
>
>         <route id="readZip" streamCache="true">
>             <from uri="file://target?fileName=any.xxx&amp;moveFailed=.error&amp;move=.done&amp;delay=5000"/>
>             <!-- <unmarshal ref="zipFileDataFormat"/> -->
>             <camel:split streaming="true">
>                 <!-- <camel:simple>${body}</camel:simple> -->
>                 <camel:ref>zipSplitter</camel:ref>
>                 <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
>                 <camel:to uri="file://target/unzipped"/>
>             </camel:split>
>             <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
>         </route>
>
> ZipSplitter seems not to have this issue. Also works for 2.13.2. which could be helpful
for you @Thomas.
>
> So from my point of view CAMEL-7577 is not RESOLVED.
>
> Best regards,
>
> -----Ursprüngliche Nachricht-----
> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Gesendet: Dienstag, 2. September 2014 14:29
> An: users@camel.apache.org
> Betreff: Re: camel 2.13.2 zipdataformat
>
> You need to build the code to make sure the snapshot is using recent code.
>
> On Tue, Sep 2, 2014 at 2:22 PM, Sven Nold <Sven.Nold@isb-ag.de> wrote:
>> Hi Claus,
>>
>> I changed to 2.13.3-SNAPSHOT but still the same issue.
>>
>> Best regards,
>>
>> Sven
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Claus Ibsen [mailto:claus.ibsen@gmail.com]
>> Gesendet: Dienstag, 2. September 2014 13:52
>> An: users@camel.apache.org
>> Betreff: Re: camel 2.13.2 zipdataformat
>>
>> Hi
>>
>> See CAMEL-7577
>>
>> On Tue, Sep 2, 2014 at 1:49 PM, Sven Nold <Sven.Nold@isb-ag.de> wrote:
>>> Hi,
>>>
>>> interesting is also that if I only have on zipEntry and define the zipDataFormat
as useIterator=false it works fine. So my guess is that ZipIterator somehow holds a lock on
the file:
>>>
>>>     <bean id="zipFileDataFormat" class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>>         <property name="usingIterator" value="false"/>
>>>     </bean>
>>>
>>>     <bean id="zipAggregationStrategy"
>>> class="org.apache.camel.processor.aggregate.zipfile.ZipAggregationStr
>>> a
>>> tegy"/>
>>>
>>>     <camelContext xmlns="http://camel.apache.org/schema/spring">
>>>         <!-- write the zip so we can process something repeatedly -->
>>>         <route id="writeZip">
>>>             <from uri="file://src/main/data?antInclude=message1.xml&amp;noop=true"/>
>>>             <aggregate strategyRef="zipAggregationStrategy" completionFromBatchConsumer="true"
eagerCheckCompletion="true" completionSize="2">
>>>                 <correlationExpression>
>>>                         <constant>true</constant>
>>>                 </correlationExpression>
>>>                 <to uri="file://target?fileName=any.xxx"/>
>>>             </aggregate>
>>>         </route>
>>>
>>>         <!-- extract the zip -->
>>>         <route id="readZip" streamCache="true">
>>>             <from uri="file://target?fileName=any.xxx&amp;moveFailed=.error&amp;move=.done&amp;delay=5000"/>
>>>             <unmarshal ref="zipFileDataFormat"/>
>>>             <!--<camel:split streaming="true">
>>>                 <camel:simple>${body}</camel:simple>
>>>                 <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
>>>                 <camel:to uri="file://target/unzipped"/>
>>>             </camel:split> -->
>>>                         <camel:to uri="log:org.apache.camel?level=INFO&amp;showAll=true&amp;multiline=true&amp;showStreams=true"/>
>>>         </route>
>>>     </camelContext>
>>>
>>> Works fine.
>>>
>>> Best regards,
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Sven Nold [mailto:Sven.Nold@isb-ag.de]
>>> Gesendet: Dienstag, 2. September 2014 13:38
>>> An: users@camel.apache.org
>>> Betreff: AW: camel 2.13.2 zipdataformat
>>>
>>> Hi,
>>>
>>> thanks for this test now I think here is the problem:
>>>
>>> [l-1) thread #1 - file://target] GenericFileOnCompletion        WARN  Error during
commit. Exchange[any.zip]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException
- Error renaming file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip
to target\.done\any.zip]
>>> org.apache.camel.component.file.GenericFileOperationFailedException: Error renaming
file from C:\temp\camel-examples-master\zip-file-processing\target\any.zip to target\.done\any.zip
>>>         at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:77)
>>>         at org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
>>>         at org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
>>>         at org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:124)
>>>         at org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:80)
>>>         at org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:54)
>>>         at org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:100)
>>>         at org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:228)
>>>         at org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:61)
>>>         at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:613)
>>>         at org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:581)
>>>         at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:240)
>>>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:106)
>>>         at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>>>         at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:423)
>>>         at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:211)
>>>         at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:175)
>>>         at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
>>>         at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
>>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>         at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.io.IOException: Renaming file from 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip'
to 'target\.done\any.zip' failed: Cannot delete file 'C:\temp\camel-examples-master\zip-file-processing\target\any.zip'
after copy succeeded
>>>         at org.apache.camel.util.FileUtil.renameFileUsingCopy(FileUtil.java:456)
>>>         at org.apache.camel.util.FileUtil.renameFile(FileUtil.java:428)
>>>         at org.apache.camel.component.file.FileOperations.renameFile(FileOperations.java:74)
>>>         ... 25 more
>>>
>>> This is why it grabs the file all the time.
>>>
>>> Regarding your questions:
>>>
>>> 1) well thanks to your test you can see error above
>>> 2) yes it is processing it over and over again
>>> 3) no doesn't move to .error
>>> 4) no file is not updated, in your test case I am not quite sure, will increase
delay or as you suggested preMove option.
>>>
>>> Due to known problems off windows and zip files I even renamed it to .xxx to
keep away explorer from analyzing, but still it seems there is a handle on that file.
>>> And again if I remove the unmarshal and split it works fine.
>>>
>>> Thanks for any help.
>>>
>>> Best regards,
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Jimmi Dyson [mailto:jimmidyson@gmail.com]
>>> Gesendet: Dienstag, 2. September 2014 10:37
>>> An: users@camel.apache.org
>>> Betreff: Re: camel 2.13.2 zipdataformat
>>>
>>> Hi,
>>>
>>> I just added an example at
>>> https://github.com/jimmidyson/camel-examples/tree/master/zip-file-processing.
>>> This example creates the zip file first (so that it's repeatable) & then
does pretty much what you're doing: read from file uri, unmarshal, log each file contents
(careful with large files) & finally writes each file to a file. Seems to be doing what
it should...
>>>
>>> Few questions if the example doesn't help you:
>>>
>>>    - Are there any hints in the logs (try increasing logging level if not)?
>>>    - If the file is back in the source dir does it try to process again?
>>>    - Is there anything in the .error (moveFailed dir)?
>>>    - Could the source file be being updated while you're processing it - if
>>>    so, can you try the preMove option?
>>>
>>> Thanks,
>>>
>>>
>>> On 2 September 2014 08:46, Walzer, Thomas
>>> <thomas.walzer@integratix.net>
>>> wrote:
>>>
>>>> Hi Sven,
>>>>
>>>> I seem to have a similar problem. I also use 2.13.2.
>>>> My code is almost identical. The only difference is my in-url, that
>>>> looks
>>>> like:
>>>>
>>>> file:////someserver/somedir/in?move=archive/${date:now:yyyyMMdd}/${f
>>>> i l e:name}.${date:now:HHmmss}&amp;antInclude=*.ZIP
>>>>
>>>> I get a random number of retrievals (about 4-20) and therefore the
>>>> same number of files in the „archive“ directory. The timestamps are
>>>> 6 seconds apart.
>>>>
>>>> Cheers, Thomas.
>>>>
>>>> Am 02.09.2014 um 08:44 schrieb Sven Nold <Sven.Nold@isb-ag.de>:
>>>>
>>>> > Hi Guys,
>>>> >
>>>> > I am trying to read Zipentries and process them separately. But
>>>> > using
>>>> the file component it seems the file is not correctly closed. I can
>>>> see the file for a few seconds in the .done and source directory.
>>>> But then it disappears in the .done folder and is back in the source directory
:
>>>> >
>>>> >
>>>> > <bean id="zipFileDataFormat"
>>>> class="org.apache.camel.dataformat.zipfile.ZipFileDataFormat">
>>>> >    <property name="usingIterator" value="true" />  </bean>
>>>> >
>>>> > <route id="readZip">
>>>> >      <from
>>>> uri="file://C:/temp/?fileName=any.zip&amp;moveFailed=.error&amp;move=.done&amp;delay=10000"
>>>> />
>>>> >      <unmarshal ref="zipFileDataFormat" />
>>>> > <!--       <split streaming="true"> -->
>>>> > <!--             <simple>$simple{body}</simple> -->
>>>> > <!--             <to
>>>> uri="log://OUT?level=INFO&amp;showAll=true&amp;multiline=true" />
>>>> -->
>>>> > <!--       </split> -->
>>>> >      <to uri="log://OUT?level=INFO&amp;showAll=true&amp;multiline=true"
>>>> />
>>>> > </route>
>>>> >
>>>> > If I remove the unmarshal the file component works fine (it moves
>>>> processed file to .done).
>>>> >
>>>> > Any Ideas? I guess I could do a workaround by using noop=true and
>>>> > moving
>>>> the file myself, but it would be nice if it works as expected.
>>>> >
>>>> > Best regards,
>>>> >
>>>> > Sven
>>>>
>>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: cibsen@redhat.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> hawtio: http://hawt.io/
>> fabric8: http://fabric8.io/
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/

Mime
View raw message