Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
Received-SPF: pass (athena.apache.org: domain of claus.ibsen@gmail.com
 designates 209.85.218.211 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=H4SA39GYeV2WXuGWluaWcsrtoEifC9jZeYrpdc90qfASx+wMVaPerhAHocW1gnzw/8
         uA46/fDv4Db7KXhcQR0jZN2cHLSgeAAHC2To1P9JO8eItLgWPgSJ+ET+cvK4HqOmB+6P
         pPf0Ai9dU3b28xz98xykyU6dEuHBShXLPlZYs=
MIME-Version: 1.0
In-Reply-To: <26535877.post@talk.nabble.com>
References: <26522808.post@talk.nabble.com>
 <5380c69c0911260301l76babc76k3f3b3504eb1c038@mail.gmail.com>
	<26535877.post@talk.nabble.com>
From: Claus Ibsen <claus.ibsen@gmail.com>
Date: Fri, 27 Nov 2009 09:18:28 +0100
Message-ID: <5380c69c0911270018h34e1b0edp8152231f7fa9005c@mail.gmail.com>
Subject: Re: Lifecycle of a Camel Component
To: users@camel.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Peter


On Fri, Nov 27, 2009 at 5:33 AM, Pete Mueller <pete@routecloud.com> wrote:
>
> Thanks very much for your answers, I also found an excellent programming
> guide at the FUSEsource site. =A0That helped a lot as well.
>
> I understand what you wrote below, but it raises some concerns in my mind
> with the reliability of Camel when dealing with unreliable external
> entities. =A0I'm guessing that just stems from a lack of understanding on=
 my
> part. =A0If it helps, I come from an EDI background where systems are des=
igned
> to gaurantee delivery of millions of messages a day to thousands of
> different endpoints. =A0I'll try and illustrate my concern.
>
> I have two routes, the input, from a custom component, is a graphic image
> that is analyized (which can take 30-90 seconds) and them placed on a que=
ue.
> The second route takes items off the queue and emails them to a destinati=
on
> AND puts the modified image back on the server.
> Something like:
>
> from("mycomp:host")
> .to(new MyImageProcessor())
> .to("amq:input_queue");
>
> from("amq:"input_queue")
> multicast().to("smtp:test@test.com", "mycomp:host");
>
> Here what I need to account for:
> 1. =A0Most importantly, I cannot lose any messages. =A0So let's say that =
a
> message is taken off the queue and passed to the smtp component, but the
> mail server is not up, so it throws an exception.
> - What happens to the message? =A0Does it get placed back on the queue? =
=A0Is
> there a concept of transactions around routes so that only after BOTH
> endpoints confirm a sucessful delivery does the message get removed from =
the
> queue?

Yes you can use transactions so the message will be rolled back to the
JMS queue.

> - What if the smtp server sucessfully delivers and "mycomp" throws an
> exception?
>

SMTP protocol does not support transactions, its in fact only a few
protocols that does this such as JMS, JDBC.
The same applies for HTTP based as well.

Its not a limitation in Camel but just in general that these protocols
cannot participate in distributed transactions.


> I am aware of the exceptionHandler() idea, but it seems exceptionHandlers
> are placed on a RouteCollection, not a single route. Each instance of
> RouteBuilder could have one route in it. =A0But that creates a lot of ext=
ra
> classes.
>

No you can have error handlers on a global and route scope.


> 2. After ensuring that i did not lose the message pending at the smtp
> component, I need to shut down the routes since the smtp service is dead
> (for the moment). =A0This would seem to be a job for exceptionHandler(),
> however that handler would need to know which object threw the exception =
(in
> this case the smtp Producer) and what route it belongs to. =A0then, it wo=
uld
> need to scan all other route that use the same endpoint, and stop them to=
o.
> I see no way of doing this from within the handler, since it is not notif=
ied
> which Producer threw the exception. =A0Also, I see no way of consistently
> getting from a Producer to it's owner endpoint.

In Camel 2.1 a header will be enriched with the endpoint that failed
so you got that information at hand now.
Exchange.FAILURE_ENDPOINT


> - How do you gracefully and without data loss bring down a route?
> - Does bringing down a route destroy the consumer/producer objects? or ju=
st
> stop() them?
>

See for example RoutePolicy. Also if you use JMS as "safe grounds" for
messages then you will not loose messages.
So if a remote SMTP server is temporary down the need to shutdown a
route is not as important.


> 3. =A0Assuming I could figure out which routes to shut down. =A0I can exe=
cute
> route.stop(). =A0But what happens to messages that are "in process" =A0ta=
ke for
> example a message from mycomp that is currently being analyzed, when the
> analyzer finishes (say 20-30 seconds after the route.stop() was called:
> - Is the queue Producer still available?
> - Will route.stop() block until all parts of the route are shut down?
> - Will route.stop() stop ONLY the consumer that is part of it's route, or
> the entire route?

We have graceful shutdown on the roadmap for 2.2. However the key
pieces are in 2.1 already.
There is an in flight registry you can poke to see if there are any
exchanges in progress on a particular route.
And thus know when to stop it.


> - Really, I would not need to bring down the route, just stop() the consu=
mer
> at the beginning. =A0Do the Producers/Consumers/Endpoint have any knowled=
ge
> about the route they are on? Or can they get it some how?
> - Can a Consumer/Producer be shared amongst several routes?
>

Yes but then you need to do that manually as the fluent builder will
create a new consumer/producer per route.
And you can design your custom endpoint to return a shared
consumer/producer if you really need this?


> It could be that I have the wrong idea of what Camel is meant to do. =A0A=
ll of
> the examples I've seen deal with short routes from some system to some ot=
her
> system where it seems the assumption is made that everything is up all th=
e
> time. =A0That is the nature of examples. =A0When dealing with Enterprise
> Integrations (well at least the integration I deal with), the biggest iss=
ue
> is always data integrity during connection failures. =A0It could be that =
all
> this transactional nature and data integrity is in Camel, I just don't se=
e
> it. =A0It is also a valid answer to say that I need to "roll my own" syst=
em,
> but that lessens the usefulness of Camel as an "Enterprise" tool.
>
> BTW, if we can figure all this out with Camel, then I'll be happy to add =
an
> example to the collection that shows how to setup message "integrity".
> -p
>

Consider looking at the ActiveMQ in Action book which covers such
scenarios as the broker is the backbone for such an infrastructure
with messaging "integrity".

There are some case studies here
http://fusesource.com/resources/collateral/

For example Sabre have been running for more than 1,5 years and not
had a downtime and lost a message.

Camel itself is just a framework/API that makes working with the
zillion of other protocols/frameworks much simpler.

And if you want to know much more about Camel and how it works, the
online documentation is of course avail.
But we are writing a book at the moment which covers many details how
it works inside Camel and whatnot.

Not trying to sell but just saying that the book should be considered
a well structured reference and user guide for Camel.
The wiki pages just dont cut it, its a bit to messy. But they are
great for searching, so just type some keywords in the searchbox on
the frontpage.


>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> A lot of questions. Let me give a start and try answering a couple of
>> those.
>>
>>
>>
>> On Thu, Nov 26, 2009 at 1:42 AM, Pete Mueller <pete@routecloud.com> wrot=
e:
>>>
>>> Does any one have any documentation on the lifecycle of a camel compone=
nt
>>> and
>>> it's associated entities (Endpoint,
>>> Consumer, Producer). =A0I am attempting to write a component for a
>>> event-driving wireline protocol. =A0I understand the basics of:
>>> 0. Component is created.
>>> 1. URI in DSL gets passed to to component.
>>> 2. Component creates Endpoint for each unique URI (if using a singleton=
)
>>> 3. Endpoint creates Consumers and Producers.
>>>
>>> But I'm looking for a little more detail as to what triggers the creati=
on
>>> and removal of say Consumers. Specifically,
>>> - Is a single Consumer around for the life of the CamelContext?
>>
>> Consumers and Producers are in reality more dynamic as you can have
>> create a consumer, use it once and then discard it.
>> CamelContext does not hold a repository for all created
>> consumers/producers etc. As you can create then without involvement
>> from CamelContext.
>>
>> On the other hand when a consumer is used as a input to a route, then
>> that consumer is around for the life of that route.
>>
>>
>>> - Does an Endpoint get notified if a Consumer it created is stop()'d?
>>
>> No there is no such callback. Camel have a very liberal and light API
>> for Endpoint, Consumer, Producer so its easy to work with an implement
>> custom components etc.
>>
>> There is a event API added in Camel 2.1 which have notifications when
>> routes are started/stopped etc. This can be used in case you need to
>> do some custom stuff.
>>
>>> - What is Consumer.stop() used for? =A0Only during shutdown of the
>>> CamelContext? =A0Or is it more like a "pause" than a stop?
>>
>> Both. You can stop a route at runtime. But when Camel shutdown it will
>> go through all routes, components, endpoints, services etc. Basically
>> all what it has and stop them one by one.
>>
>>
>>> - Is there a way (say in the case of a loss of network connection) to
>>> stop()
>>> an Endpoint and all associated Consumers and Producers? Then later
>>> restart
>>> them after connection has been regained?
>>
>> Yeah you can for example use Camel error handling and stop a route in
>> case of an ConnectionException or what you like.
>>
>>
>> In Camel 2.1 you can use RoutePolicy to that as well. However you must
>> implement some logic yourself that monitors that particular connection
>> and can start/stop the route depending on its state. Camel does not do
>> that for you.
>>
>>
>>> - Is there a general example/recommendation on how to handle connection=
s
>>> to
>>> unreliable external entities?
>>>
>>
>> No not 2 business are the same. Some wants to self heal and just try
>> at next interval. Others want to try 5 times with X delays and then
>> alert someone if still a problem.
>>
>>
>>> I've been using the IRC component as somewhat of a model, but my glance
>>> through the code doesn't really indicate how the component deals with
>>> connection loss or net-splits and other networking pit-falls.
>>>
>>
>> Again connectivity is not that easy to encompass into a single model
>> for dealing with lost connections.
>> Often you only got the wacky java Exceptions that may or may not
>> indicate a io/connection problem.
>>
>> However when you have a Camel consumer =3D input to a route then you got
>> a chicken egg problem as Camel does not take over
>> before the consumer has successfully got a message and created an
>> Exchange to be routed in Camel. At this point Camel take over and you
>> can use Camel error handling. But before that all error handling etc.
>> is component specify.
>>
>>
>> We love contributions so dig into it and help out.
>> http://camel.apache.org/contributing.html
>>
>>
>>> Here's a short outline of the wire protocol I have to implement if it
>>> helps:
>>>
>>> 1. Open TWO TCP connections with Server
>>> 2. Register Myself with Server, twice, once as a SENDER, once as a
>>> RECEIVER.
>>> Both connections are bi-directional, but the functions I can do on each
>>> connection is limited by how I register myself.
>>> 3. At this point I will receive data packets (messages) for some length
>>> of
>>> time over the Receiver connection.
>>> 4. At some point in the future the Server will send me an "ALL DONE"
>>> message. =A0At which point I must un-register my sender and receiver, b=
ut
>>> the
>>> TCP connections remail open.
>>> 5. After the unregister I need to notify Consumers that there is no mor=
e
>>> messages coming and stop Producers from attempting to send more message=
s.
>>> 6. After waiting X seconds (X in most cases is 180) I will attempt to
>>> re-register myself and if successful need to re-activate Consumers and
>>> Producers.
>>> 7. I also need to account for the fact that at any time, I may lose a
>>> network connection to the Server, and will need to stop all Consumers a=
nd
>>> Producers until connection is restored.
>>>
>>> Thanks for any help
>>> -p
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Lifecycle-of-a-Camel-Component-tp26522808p2652280=
8.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Lifecycle-of-a-Camel-=
Component-tp26522808p26535877.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--=20
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus