camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: camel casts exception when there is nothing to fetch from the ftp endpoint
Date Fri, 27 Nov 2009 12:41:16 GMT
On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me@farhad.eu> wrote:
>
> Wau... thanks for that. I don't understand how I could miss it, cause i
> searched a lot for such a feature.
>
> Another thing that I have experienced in 2.0, is that the "file" component
> is unable to move/copy files larger than 2Gigs, such large files are cut
> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the OS
> or the filesystems or what. I have seen it both in Linux (debian) and Mac OS
> X. And I don't understand either, why the file component does not use the
> NIO api or at least we don't have an option to tell it use the NIO api
> instead.
> Anyway, for now I've solved the problem by using a smiple component instead
> of the "file" comp. to move/copy files using the NIO api. It is much faster
> and it has no problem with very large files.
>

For move it uses java.io.File rename and are you saying its not
optimized for the underlying OS?
I would assume that it could move a 2gb file in no time if the file is
on the same share.

Can you share your code that you are using?


> Bests,
> Farhad S.
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me@farhad.eu> wrote:
>>>
>>> Hi,
>>> Ok, I guess I should rather provide an AggregationStrategy.
>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>> starts
>>> to fetch everything that is on the ftp in the first call and that's not
>>> what
>>> i want, because I'm processing files over 3 Gigs and there are many of
>>> them.
>>> Then I figured that by using a timer, i could have ftp to download one
>>> file
>>> at a time!!
>>> What I basically need is to somehow tell the ftp, how many files I want
>>> to
>>> fetch at a time. This is probably doable now in camel-2.1 by using the
>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>> Do you have any suggestions?
>>
>> Yeah the FTP component extends the File component so you got all the
>> options from the file component as well
>> http://camel.apache.org/file2.html
>>
>> And there is a maxMessagesPerPoll you can use to tell how many files
>> you want at most to poll. For example = 1 in your use.
>>
>> And that option is avail in 2.0 also.
>>
>> In fact all these are as they are batch polling consumers
>> http://camel.apache.org/batch-consumer.html
>>
>>
>>
>>>
>>> Bests,
>>> Farhad S.
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> BTW why are you using a timer -> ftp when the ftp have a build in timer?
>>>>
>>>>
>>>>
>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me@farhad.eu> wrote:
>>>>>
>>>>> Hi,
>>>>> Please consider the following route:
>>>>>
>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>> 10000).
>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>
>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>>> nothing
>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now
in
>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
>>>>> =================
>>>>> 12:37:06,899 INFO  [Tracer] >>>  -->
>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>> Cannot store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> 12:37:06,899 INFO  [Tracer] >>>  -->
>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>> Cannot store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>> Cannot
>>>>> store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>        at
>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>        at
>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>        at
>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>        at
>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>        at
>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>        at
>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>        at
>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>        at
>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>        at
>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>        at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>        at
>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>        at
>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>        at
>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>        at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>        at
>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>        at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>        at
>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>        at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>        at
>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>        at
>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>        at
>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>        at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>        at
>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>        at
>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>        at
>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>        at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>        at
>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>        at
>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>        at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>        at java.util.TimerThread.run(Timer.java:462)
>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body available
>>>>> of
>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>>> type
>>>>> converter available to convert from type: null to the required type:
>>>>> java.io.InputStream with value null on the exchange: Exchange[Message:
>>>>> [Body
>>>>> is null]]
>>>>>        at
>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>        at
>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>        at
>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>        ... 29 more
>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
>>>>> converter available to convert from type: null to the required type:
>>>>> java.io.InputStream with value null
>>>>>        at
>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>        at
>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>        ... 31 more
>>>>> ============
>>>>> Now, I can remedy the situation by a try-catch block or an interceptor,
>>>>> but
>>>>> in my opinion this should be considered a bug in camel and it should
>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>
>>>>> Bests,
>>>>> Farhad Shiraz
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message