camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jnupponen <jussi.nuppo...@gmail.com>
Subject Quartz2 problem with AsyncDelayedDelivery
Date Mon, 13 Jul 2015 09:18:32 GMT
Hi!

I've run into a problem with camel-quartz2 (at least on 2.13.2 and 2.15.2)
cron scheduling and AsyncDelayedDelivery-option in the DefaultErrorHandler.
If I want to shutdown/restart routes during DefaultErrorHandlers
redeliveries quartz scheduling gets deadlocked and quietly fails all future
fire times. Without AsyncDelayedDelivery-option this problem does not occur.
I'm running my routes in bundles in Karaf 3.0.3 but the problem also occurs
when running the code on jvm (at least on 1.7) without Karaf.

I have configured quite long redelivery times for my routes (e.g. 8 times in
2 hour interval) but at times I would need to be able to restart bundles
running in Karaf even when there would be redeliveries still left (e.g. when
installing new version of the bundle).

This seems like a bug to me but what's your take on this? I've attached few
lines of log and some example route code to the end of this message.

Best regards,
- Jussi



Logs on org.quartz debug level show every few seconds:
[artzScheduler-camel-1_Worker-2] SimpleThreadPool               DEBUG
Waiting for thread DefaultQuartzScheduler-camel-1_Worker-2 to shut down

and jstack shows for DefaultQuartzScheduler-camel-1_Worker-2:
"DefaultQuartzScheduler-camel-1_Worker-2" prio=10 tid=0x00007fd2f8201000
nid=0x2a16 in Object.wait() [0x00007fd3039dc000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007ae410420> (a java.lang.Object)
	at org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:371)
	- locked <0x00000007ae410420> (a java.lang.Object)
	at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:740)
	at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:206)
	at
org.apache.camel.component.quartz2.QuartzComponent.doStop(QuartzComponent.java:352)
	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
	at org.apache.camel.util.ServiceHelper.stopService(ServiceHelper.java:141)
	at
org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:204)
	at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2827)
	at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2852)
	at
org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:2840)
	at
org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:2704)
	- locked <0x00000007ad1abd88> (a org.apache.camel.impl.DefaultCamelContext)
	at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:102)
	at
com.example.route.MyAsyncDelayedDeliveryTesting$1.process(MyAsyncDelayedDeliveryTesting.java:43)
	at
org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
	at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
	at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
	at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
	at
org.apache.camel.processor.loadbalancer.QueueLoadBalancer.process(QueueLoadBalancer.java:44)
	at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
	at
org.apache.camel.processor.loadbalancer.LoadBalancerSupport.process(LoadBalancerSupport.java:87)
	at org.apache.camel.component.quartz2.CamelJob.execute(CamelJob.java:56)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
	- locked <0x00000007ae539450> (a java.lang.Object)


Example of this situation looks like this:

package com.example.route;

import static org.apache.camel.LoggingLevel.ERROR;
import static org.apache.camel.LoggingLevel.WARN;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.joda.time.DateTime;

public class MyAsyncDelayedDeliveryTesting extends RouteBuilder {
	
		
	@Override
	public void configure() throws Exception {

		// Set lower timeout for convenience (default 300 seconds).
		getContext().getShutdownStrategy().setTimeout(10);

		// Start some route1 cron scheduled.
		DateTime now = DateTime.now().plusSeconds(2);
		from("quartz2://my_route_quartz_timer?cron="
				+now.getSecondOfMinute()+"+"
				+now.getMinuteOfHour()+"+"
				+now.getHourOfDay()+"+*+*+?")
		.id("scheduling1")
		.to("direct://my-route");
		
		
		// Start other route that shutdowns the context thus shutting down the
route1.
		// Simulate the situation where you want to restart your route when there
are still
		// ongoing redelivery tasks.
		now = DateTime.now().plusSeconds(4);
		from("quartz2://my_shutdown_route_quartz_timer?cron="
				+now.getSecondOfMinute()+"+"
				+now.getMinuteOfHour()+"+"
				+now.getHourOfDay()+"+*+*+?")
		.id("scheduling2")
		.process(new Processor() {
			
			@Override
			public void process(Exchange arg0) throws Exception {
				getContext().stop();
				
			}
		});
		
		// Do some logic that causes exception and redeliveries for 1 minute.
		from("direct://my-route")
		.id("route1")
		.errorHandler(defaultErrorHandler()
			.maximumRedeliveries(10)
			.redeliveryDelay(1000*6)
			// asyncDelayedRedelivery causes the problem. 
			.asyncDelayedRedelivery()
			.retryAttemptedLogLevel(WARN)
			.retriesExhaustedLogLevel(ERROR))

		.process(new Processor() {
			
			@Override
			public void process(Exchange exchange) throws Exception {
				throw new RuntimeException("Error");
			}
		});	
	}
}

Full example is visible here:
https://github.com/jnupponen/camel-quartz2-asyncDelayedDelivery-problem



--
View this message in context: http://camel.465427.n5.nabble.com/Quartz2-problem-with-AsyncDelayedDelivery-tp5769210.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message