Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (nike.apache.org: domain of claus.ibsen@gmail.com
 designates 209.85.218.211 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=NcmdKBGnHEkMzlF2NpZGJSQ6LqJoFJI0mjQgEQpDlCzMDj07OYGKRiW6Vf4ITdO8NB
         5F0+KBRQq+mALuKQE2s3pdPifzDcDX0cWUz6CJIkGUVxCGOn9LbcyDn9lgKgVBkGtWmE
         hAGtQIDQuwG/UFXNl/FVcH+dAAeRAvHCkb24U=
MIME-Version: 1.0
In-Reply-To: <4B2ED49A.1000501@gmail.com>
References: <5380c69c0912200234l4c379dadteb5a2d0a8ee754fe@mail.gmail.com>
	<26865006.post@talk.nabble.com> <4B2ED49A.1000501@gmail.com>
From: Claus Ibsen <claus.ibsen@gmail.com>
Date: Mon, 21 Dec 2009 07:10:50 +0100
Message-ID: <5380c69c0912202210o7a7f3567m195746b6aaa1324@mail.gmail.com>
Subject: Re: [NEW] Camel 2.2 - Graceful Shutdown - Feedback welcome
To: users@camel.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Dec 21, 2009 at 2:51 AM, Willem Jiang <willem.jiang@gmail.com> wrot=
e:
> Hi,
>
> I just have another question.
> Does the batching consumer support the transaction?
> If so we need to consider to add some option like to rollback the complet=
ed
> transactions or just let them be.
>

Its only a few components which supports "real" transactions such as
JDBC and JMS based.
And its Spring TX that does the work behind the scenes.

The batch consumers, just have the batch name as they work on a batch
of input messages, such as X files, X emails etc.
And then go over these X files one by one. Each "one" is a isolated
Exchange and thus not transacted in a big batch.
So there are no problem stopping in the middle of this. Just that
sometimes I may be "nicer" to complete this batch before shutting
down.

Suppose its a job that runs once a day, and there are like 5 files
left to go over, then it may be nice if it could give it time to work
those 5 files before shutting down.

And the Aggregator EIP does in fact have a *batchConsumer* option
which allows it to expect the "number of messages from the batch" so
if you use this option and shutdown, then the Aggregator will not be
able to complete.

Since its a graceful shutdown then I do think it would be nice and
fair to let Camel complete those batches. However its only if those
batches have a significant number of pending messages it could take
long time. Suppose a file consumer is picking up files once a day and
there are 10000 files to process :). A better example may be some
process that download big big FTP files and there are plenty of those
files.

We could also consider adding some VETO to shutting down in case all
inflight and pending messages cannot be done in due time. But that
does not go well with OSGi if you stop a camel bundle. Then you cannot
VETO this in OSGi as well. The only remedy is to give it more time to
complete all those messages.

Okay that may be going to far. Lets keep the first installment of
graceful shutdown simple :)


> Willem
>
> Ashwin Karpe wrote:
>>
>> Hi Claus,
>>
>> The ability to associate a user controlled graceful shutdown strategy is
>> very good and certainly very valuable.
>> =A0+1 for this nice feature.
>>
>> As for the question on batching consumers, it would be great to have a
>> flag(shutdownPolicy) that specifies which way a user would like the
>> endpoint
>> to behave in the event of context shutdown. Ideally this could be an
>> operation on the Batch Consumer interface which could be implemented by
>> all
>> implementing endpoints and influenced based on a URI query setting on al=
l
>> endpoints implementing Batch Consumer.
>> Just a thought....
>>
>> Cheers,
>>
>> Ashwin...
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> I am posting this to the user forum of Camel to a broader audience.
>>>
>>> I have just added Graceful Shutdown to Camel 2.2 (trunk code will be
>>> committed shortly).
>>> https://issues.apache.org/activemq/browse/CAMEL-1483
>>>
>>> The wiki page is here
>>> http://cwiki.apache.org/confluence/display/CAMEL/Graceful+Shutdown
>>>
>>> I have enabled it by default on trunk, which means the shutdown
>>> process is altered.
>>> You can read all about it on the wiki page.
>>>
>>> As its new code I would love feedback and having people giving it a
>>> test run on your systems.
>>>
>>> There is one note, the SEDA/VM components uses an internal in memory
>>> queue with pending exchanges to be processed.
>>> The graceful shutdown will take this into account and wait until all
>>> those pending messages have been run to completion.
>>> The progress is logged at INFO level such as shown on the wiki page.
>>>
>>> If there are other Camel components which have the same behavior,
>>> please let me know, as I could not remember any other that did.
>>> If so we need to let its consumer implements the ShutdownAware
>>> interface which ensures the graceful shutdown can work with it.
>>>
>>> I have set a default timeout of 300 seconds for the hole shutdown
>>> process. During this time any pending and inflight exchanges should be
>>> run to completion.
>>> Camel will of course stop input consumers before hand which ensures no
>>> new messages come into Camel. And therefore its just a matter of
>>> waiting until
>>> all those existing messages runs to completion. If you think this
>>> timeout should have a different default value, then speak up.
>>> What happens if the timeout is hit, then Camel will do a shutdown now,
>>> where it forces all routes to shutdown and shutdown the remainder
>>> services.
>>> This ensures that Camel will not block forever during shutdown.
>>>
>>>
>>> I have one questions although
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>
>>> We have a batch consumer concept which allows a consumer to batch up a
>>> number of exchanges to be routed.
>>> For example: file, ftp, mail, jpa and others implement this.
>>>
>>> The current gracful shutdown behavior is to stop during this batch.
>>> What happens is that lets say there are 23 messages in the batch and
>>> it has completed 7 of those.
>>> Then a shutdown is issued. The batch consumer will complete its
>>> current inprogress, and therefore it has completed 8. There are still
>>> 15 left in the batch. What happens
>>> is that it will break and not complete the batch.
>>>
>>> Do you think we should offer a configuration which allows the batch
>>> consumers to control their behavior to
>>> - stop as now during a batch
>>> - continue to complete the batch before shutting down
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>>
>> -----
>> --- Ashwin Karpe, Principal Consultant, PS - Opensource Center of
>> Competence Progress Software Corporation
>> 14 Oak Park Drive
>> Bedford, MA 01730
>> --- +1-972-304-9084 (Office) +1-972-971-1700 (Mobile) ---- Blog:
>> http://opensourceknowledge.blogspot.com/
>>
>>
>
>


--=20
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus