camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Quartz schedule too many workers in route
Date Thu, 24 Mar 2016 07:52:06 GMT
Hi

Have not had time to look into this. You are welcome to log a JIRA so
we can improve this in a future release
http://camel.apache.org/support.html

On Wed, Mar 23, 2016 at 5:33 PM, Hans Orbaan <hans.orbaan@docdata.eu> wrote:
> Hi,
>
> The only way I have found to make sure a file or ftp component does not use multiple
threads while consuming very large batches is to annotate the QuartzScheduledPollConsumerJob
class with @DisallowConcurrentExecution.
>
> That’s it. It seems to work for us. I am not familiar enough with the Camelcode to
say what side effects it has and if this would prevent any quartz job in camel to now be single
threaded, even if the user does not want it to be.
> But to me it looks like an oversight when moving from quartz to quartz2. A file or ftp
consumer should be single threaded while retrieving.
>
> Hope this helps...
>
> With kind regards,
>
> Hans Orbaan
>
>
> -----Oorspronkelijk bericht-----
> Van: Hans Orbaan [mailto:hans.orbaan@docdata.eu]
> Verzonden: Wednesday 23 March 2016 9:52
> Aan: users@camel.apache.org
> Onderwerp: RE: Quartz schedule too many workers in route
>
> Hi Claus,
>
> Did you have a chance to check this already?
> I really think file and ftp consumers should never use multiple threads unless explicitly
defined. This means a from(file://) with a quartz schedule should be stateful by default.
Only then would it have the same behaviour as spring scheduler or when using a delay.
>
> Thanks!
>
> With kind regards,
>
> Hans Orbaan
>
>
> -----Oorspronkelijk bericht-----
> Van: Hans Orbaan [mailto:hans.orbaan@docdata.eu]
> Verzonden: Monday 21 March 2016 16:52
> Aan: users@camel.apache.org
> Onderwerp: RE: Quartz schedule too many workers in route
>
> Hi Claus,
>
> It was in the email I forwarded. So below my new email. But here it is:
>
> from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter=#fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler.cron=0+0/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerId=nl_bij-export-to-archive-276)
> to(file:///data/work/sl/work-archive/work/276/)
>
> But the same goes for any component used it looks like (ftp, sftp, etc)
>
> Thanks!!
>
> With kind regards,
>
> Hans Orbaan
>
> -----Oorspronkelijk bericht-----
> Van: Claus Ibsen [mailto:claus.ibsen@gmail.com]
> Verzonden: Monday 21 March 2016 16:37
> Aan: users@camel.apache.org
> Onderwerp: Re: Quartz schedule too many workers in route
>
> On Mon, Mar 21, 2016 at 3:23 PM, Hans Orbaan <hans.orbaan@docdata.eu> wrote:
>> Hi,
>>
>> Is there someone that can help a bit with the issue below?
>>
>> An endpoint (file) with a quartz cronschedule is not stateful. This means an schedule
that triggers every minute it will start processing with multiple threads after 1 minute if
it did not finish the job. My quess is this is unwanted in almost all situations.
>>
>> This issue does not happen when spring is used as scheduler. Also this does not happen
when using a regular delay=1m.
>>
>> I tried stuff like job.stateful=true, but that is not an allowed option.
>> At this point I see no other solution then to start using the timer or quartz component
to trigger routes, but I would much rather keep using the file, ftp, etc components with a
schedule.
>>
>
> Ah can you provide an example of the uri you configure for this. We could look into what
it would take to make scheduler.job.stateful=true or something being supported.
>
>
>> Any help would be appreciated.
>>
>> With kind regards,
>>
>> Hans Orbaan
>>
>> -----Oorspronkelijk bericht-----
>> Van: Hans Orbaan [mailto:hans.orbaan@docdata.eu]
>> Verzonden: Wednesday 16 March 2016 10:18
>> Aan: users@camel.apache.org
>> Onderwerp: Quartz schedule too many workers in route
>>
>> Hello all,
>>
>> We have a few very busy routes that are throwing FileNotFoundExceptions when processing
files (full stacktrace below). Searching for those filenames in the log it is very clear that
those files have already been processed in the past minute and thus are not there.
>> After removing the 1 minute schedule and adding a 1 minute delay the problems have
disappeared.
>>
>> After testing it looks like a cron scheduler can only start 1 thread/worker per route
and a next scheduler ping will wait for the previous one to be finished. Even if we manually
trigger the route by its triggerId. This was tested by holding a thread in a debugger for
a few minutes, no new threads passed the breakpoint until we let the first thread go. Still
we are seeing what we are seeing but I cannot create the volume of messages and routes in
a test/develop environment.
>>
>> During low volume processing we can see multiple workers busy, but always in a different
minute then the other. So 1 thread/worker is busy per minute but the worker name can differ.
>> During high volume processing at some point multiple workers will be processing,
the same route, in the same second. Also there have been moments we can find 5 camellock files
at the same time, in the same directory. There is only 1 route per directory so there should
not be more than 1 camellock, I think.
>>
>> Example from log (all the same routeId):
>> 2016-03-14 23:35:38,783 [DefaultQuartzScheduler-esbContext-1_Worker-1]
>> 2016-03-14 23:35:40,385 [DefaultQuartzScheduler-esbContext-1_Worker-7]
>> 2016-03-14 23:35:40,620 [DefaultQuartzScheduler-esbContext-1_Worker-6]
>> 2016-03-14 23:35:40,843 [DefaultQuartzScheduler-esbContext-1_Worker-8]
>> 2016-03-14 23:35:41,262 [DefaultQuartzScheduler-esbContext-1_Worker-1]
>> 2016-03-14 23:35:41,529 [DefaultQuartzScheduler-esbContext-1_Worker-2]
>> 2016-03-14 23:35:43,200 [DefaultQuartzScheduler-esbContext-1_Worker-7]
>> 2016-03-14 23:35:43,205 [DefaultQuartzScheduler-esbContext-1_Worker-6]
>> 2016-03-14 23:35:44,470 [DefaultQuartzScheduler-esbContext-1_Worker-8]
>> 2016-03-14 23:35:44,665 [DefaultQuartzScheduler-esbContext-1_Worker-1]
>> 2016-03-14 23:35:45,503 [DefaultQuartzScheduler-esbContext-1_Worker-2]
>> 2016-03-14 23:35:46,393 [DefaultQuartzScheduler-esbContext-1_Worker-7]
>> 2016-03-14 23:35:47,051 [DefaultQuartzScheduler-esbContext-1_Worker-6]
>>
>> The routes are very simple, from a mounted folder to a local directory with nothing
in between except for some onException handling for retrying and sending to an error route
(direct). Lowering maxMessagesPerPoll (250) did seem to decrease the problem a bit but I am
not convinced yet. This should be tested with a number lower than 100.
>>
>> from(file:///mnt/sl-nl/bij/outbox/?sortBy=ignoreCase:file:name&filter=
>> #fileFilter&recursive=false&move=processed&moveFailed=failed&scheduler
>> .cron=0+/1+0-23+?+*+1,2,3,4,5,6,7&scheduler=quartz2&scheduler.triggerI
>> d=nl_bij-export-to-archive-276)
>> to(file:///data/work/sl/work-archive/work/276/)
>>
>> Is there anything we can change to make sure we have only one worker processing at
a time, per route? We seem to have 10 workers total that are divided amongst the routes with
schedulers. We have 250 routes total with a schedule. Most schedules are every minute. Also
there are 250 routes with delay=5s, but they do not use those workers.
>>
>> org.quartz.threadPool.threadCount is now at 1. We might need to increase this number
as we have 250 routes with a 1 minute schedule. But the issue we have is quite the opposite
of what to expect if quartz cannot trigger all jobs it should trigger.
>>
>> -----------------------------------------
>> version.camel   2.16.2
>>
>> file:quartz.properties
>> org.quartz.scheduler.instanceName = QuartzScheduler
>> org.quartz.scheduler.threadsInheritContextClassLoaderOfInitializer =
>> true org.quartz.scheduler.skipUpdateCheck = true
>> org.quartz.scheduler.jmx.export = true org.quartz.threadPool.class =
>> org.quartz.simpl.SimpleThreadPool org.quartz.threadPool.threadCount =
>> 1 org.quartz.jobStore.class = org.quartz.simpl.RAMJobStore
>>
>> server.info
>> Tomcat Version                               Apache Tomcat/8.0.26
>> JVM Version                                     1.8.0_60-b27
>> JVM Vendor                                     Oracle Corporation
>> OS Name                                            Linux
>> OS Version                                        3.8.13-98.2.2.el6uek.x86_64
>> OS Architecture                               amd64
>>
>> 2016-03-14 22:29:02,485 [DefaultQuartzScheduler-esbContext-1_Worker-10] errors -
WARN - Error stacktrace: org.apache.camel.component.file.GenericFileOperationFailedException:
Cannot store file: /data/work/sl/work-archive/work/276/DD_SOS2016031422110816.xml
>>         at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:292)
>>         at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:277)
>>         at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
>>         at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:79)
>>         at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
>>         at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
>>         at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
>>         at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
>>         at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
>>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
>>        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>>         at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442)
>>         at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214)
>>         at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178)
>>         at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
>>         at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
>>         at org.apache.camel.pollconsumer.quartz2.QuartzScheduledPollConsumerJob.execute(QuartzScheduledPollConsumerJob.java:59)
>>         at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
>>         at
>> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.ja
>> va:573) Caused by: org.apache.camel.InvalidPayloadException: No body
>> available of type: java.io.InputStream but has value: GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]
of type: org.apache.camel.component.file.GenericFile on: DD_SOS2016031422110816.xml. Caused
by: Error during type conversion from type: java.lang.String to the required type: byte[]
with value [Body is file based: GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]]
due java.io.FileNotFoundException: /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such
file or directory). Exchange[ID-slesb101-60348-1457682250340-0-642502][DD_SOS2016031422110816.xml].
Caused by: [org.apache.camel.TypeConversionException - Error during type conversion from type:
java.lang.String to the required type: byte[] with value [Body is file based: GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]]
due java.io.FileNotFoundException: /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such
file or directory)]
>>         at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>         at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:273)
>>         ... 19 more
>> Caused by: org.apache.camel.TypeConversionException: Error during type conversion
from type: java.lang.String to the required type: byte[] with value [Body is file based: GenericFile[/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml]]
due java.io.FileNotFoundException: /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such
file or directory)
>>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:610)
>>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:177)
>>         at org.apache.camel.component.file.FileBinding.loadContent(FileBinding.java:57)
>>         at org.apache.camel.component.file.GenericFileConverter.genericFileToInputStream(GenericFileConverter.java:123)
>>         at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:497)
>>         at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243)
>>         at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
>>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:293)
>>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168)
>>         at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:99)
>>         ... 20 more
>> Caused by: org.apache.camel.RuntimeCamelException: java.io.FileNotFoundException:
/mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml (No such file or directory)
>>         at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1642)
>>         at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1247)
>>         at org.apache.camel.impl.converter.StaticMethodTypeConverter.convertTo(StaticMethodTypeConverter.java:59)
>>         at org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:97)
>>         at sun.reflect.GeneratedMethodAccessor555.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:497)
>>         at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243)
>>         at org.apache.camel.impl.converter.StaticMethodFallbackTypeConverter.convertTo(StaticMethodFallbackTypeConverter.java:62)
>>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:333)
>>         at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:168)
>>         ... 30 more
>> Caused by: java.io.FileNotFoundException: /mnt/sl-nl/bij/outbox/DD_SOS2016031422110816.xml
(No such file or directory)
>>         at java.io.FileInputStream.open0(Native Method)
>>         at java.io.FileInputStream.open(FileInputStream.java:195)
>>         at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>         at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:78)
>>         at org.apache.camel.converter.IOConverter.toByteArray(IOConverter.java:266)
>>         at sun.reflect.GeneratedMethodAccessor604.invoke(Unknown Source)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>        at java.lang.reflect.Method.invoke(Method.java:497)
>>         at org.apache.camel.util.ObjectHelper.invokeMethod(ObjectHelper.java:1243)
>>         ... 39 more
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Mime
View raw message