camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: camel casts exception when there is nothing to fetch from the ftp endpoint
Date Fri, 27 Nov 2009 13:30:34 GMT
On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me@farhad.eu> wrote:
>
> That's probably the case. I've tried with different buffer sizes (though not
> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on
> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
> That would be great, if one could specify the buffer-size for the "file"
> component.
>

You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
there something that play tricks on me?

The file component already have a bufferSize option with a default of 128kb
http://camel.apache.org/file2.html

So you can use that to set what you want to use.



> -Farhad S.
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me@farhad.eu> wrote:
>>>
>>> Sure. I have attached the file.
>>> Actually I use from->to components for transfering files and I guess it
>>> copies the file, rather than moving.
>>>
>>
>> Ah makes more sense for copying to leverage NIO.
>>
>> Camel does this also but I can see that it may hit that 2gb limit as
>> it doesnt transfer in a loop using a lower buffer.
>>
>> It was doing it in a one liner
>>             in.transferTo(0, in.size(), out);
>>
>> I wonder if you really must do that in a loop to avoid any 2gb limit
>> or the likes?
>>
>>
>>> Thanks,
>>> Farhad S.
>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me@farhad.eu> wrote:
>>>>>
>>>>> Wau... thanks for that. I don't understand how I could miss it, cause
i
>>>>> searched a lot for such a feature.
>>>>>
>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>> component
>>>>> is unable to move/copy files larger than 2Gigs, such large files are
>>>>> cut
>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with
>>>>> the
>>>>> OS
>>>>> or the filesystems or what. I have seen it both in Linux (debian) and
>>>>> Mac
>>>>> OS
>>>>> X. And I don't understand either, why the file component does not use
>>>>> the
>>>>> NIO api or at least we don't have an option to tell it use the NIO api
>>>>> instead.
>>>>> Anyway, for now I've solved the problem by using a smiple component
>>>>> instead
>>>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>>>> faster
>>>>> and it has no problem with very large files.
>>>>>
>>>>
>>>> For move it uses java.io.File rename and are you saying its not
>>>> optimized for the underlying OS?
>>>> I would assume that it could move a 2gb file in no time if the file is
>>>> on the same share.
>>>>
>>>> Can you share your code that you are using?
>>>>
>>>>
>>>>> Bests,
>>>>> Farhad S.
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me@farhad.eu> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>> The reason why I use a timer is that when I use the ftp endpoint,
it
>>>>>>> starts
>>>>>>> to fetch everything that is on the ftp in the first call and
that's
>>>>>>> not
>>>>>>> what
>>>>>>> i want, because I'm processing files over 3 Gigs and there are
many
>>>>>>> of
>>>>>>> them.
>>>>>>> Then I figured that by using a timer, i could have ftp to download
>>>>>>> one
>>>>>>> file
>>>>>>> at a time!!
>>>>>>> What I basically need is to somehow tell the ftp, how many files
I
>>>>>>> want
>>>>>>> to
>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by
using
>>>>>>> the
>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>> Do you have any suggestions?
>>>>>>
>>>>>> Yeah the FTP component extends the File component so you got all
the
>>>>>> options from the file component as well
>>>>>> http://camel.apache.org/file2.html
>>>>>>
>>>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>
>>>>>> And that option is avail in 2.0 also.
>>>>>>
>>>>>> In fact all these are as they are batch polling consumers
>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Bests,
>>>>>>> Farhad S.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> BTW why are you using a timer -> ftp when the ftp have
a build in
>>>>>>>> timer?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me@farhad.eu>
wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Please consider the following route:
>>>>>>>>>
>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>> 10000).
>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>
>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0,
if there
>>>>>>>>> is
>>>>>>>>> nothing
>>>>>>>>> to fetch from the ftp. I got rid of the exception by
a
>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around
the route.
>>>>>>>>> Now
>>>>>>>>> in
>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on
the "to"
>>>>>>>>> endpoint:
>>>>>>>>> =================
>>>>>>>>> 12:37:06,899 INFO  [Tracer] >>>  -->
>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>> Cannot store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> 12:37:06,899 INFO  [Tracer] >>>  -->
>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>> Cannot store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>> Cannot
>>>>>>>>> store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>        at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException:
No body
>>>>>>>>> available
>>>>>>>>> of
>>>>>>>>> type: java.io.InputStream on: Message: [Body is null].
Caused by:
>>>>>>>>> No
>>>>>>>>> type
>>>>>>>>> converter available to convert from type: null to the
required
>>>>>>>>> type:
>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>> Exchange[Message:
>>>>>>>>> [Body
>>>>>>>>> is null]]
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>        ... 29 more
>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException:
No
>>>>>>>>> type
>>>>>>>>> converter available to convert from type: null to the
required
>>>>>>>>> type:
>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>        at
>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>        ... 31 more
>>>>>>>>> ============
>>>>>>>>> Now, I can remedy the situation by a try-catch block
or an
>>>>>>>>> interceptor,
>>>>>>>>> but
>>>>>>>>> in my opinion this should be considered a bug in camel
and it
>>>>>>>>> should
>>>>>>>>> silently ignore if there was nothing to fetch from the
ftp.
>>>>>>>>>
>>>>>>>>> Bests,
>>>>>>>>> Farhad Shiraz
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message