camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: ThrottlingInflightRoutePolicy does not take effect in DSL
Date Sun, 29 Nov 2009 14:38:29 GMT
On Sun, Nov 29, 2009 at 3:27 PM, shirazi <me@farhad.eu> wrote:
>
> Well, ok, I guess the "inflight exchanges" is the keyword here...
>

Yes its "inflight" so that means currently being processed inside Camel.

You are 100% free to implement your own logic and whatever in your own
RoutePolicy.
As I read your use case it appears to be based on number of files in
"processed" folders.

So just do a logic which does traverse those folders and count the
number of files.
And then if that count > your threshold then suspend the route.

You can take a look at how the existing route policy is implemented
and do something like that but instead of using "in flight exchanges"
then use that file count instead.


> Thanks,
> Farhad
>
>
> shirazi wrote:
>>
>> Hmmm... ok. Then i don't get what the scope attribute of the RoutePolicy
>> should mean. The doc. says:
>> "scope (Route | Context) : A scope for either Route or Context which
>> defines if the current number of inflight exchanges is context based or
>> for that particular route."
>> I would expect that by defining the scope as "Context" then the total
>> number of exchanges in all the routes defined in the context would be
>> taken into account by the throttling policy.
>>
>> -Farhad
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Sun, Nov 29, 2009 at 2:57 PM, shirazi <me@farhad.eu> wrote:
>>>>
>>>> Well, the example I've depicted is a simplified version of a much more
>>>> complicated flow. The scenario i've tried to test is such that: there
>>>> are a
>>>> couple of hundred files in the start folder, and I want to ensure that
>>>> there
>>>> are max. 3 files in all the subroutes in the context at any given time.
>>>> So
>>>> what i expect is that after appr. 3-4 seconds there would be 3 files in
>>>> proceed1 and proceed2 folders, and before at least one of them end up in
>>>> the
>>>> archive folder (would take 20 seconds), the throttling policy should
>>>> stop
>>>> the file consumer to read more files in and hence no files should be
>>>> coming
>>>> in the proceed folders in the meantime.
>>>> But it is not what's happening. The files keep coming into the proceed
>>>> folders and after 20 seconds I have around 20 files in the proceed
>>>> folders
>>>> ??! Do you have any idea?
>>>>
>>>
>>> Well its metrics used for the throttling are *in flight* which means
>>> Exchange objects currently inside CamelContext being routed (in the
>>> JVM memory).
>>> When an Exchange is done (= complete) such as in your case when it
>>> have been written to a proceed folder, then that Exchange is removed
>>> from memory,
>>> and therefore also do no longer count as an *in flight* Exchange.
>>>
>>> You have to create your own logic if the throttling should be based on
>>> number of files listed in the proceed folders.
>>>
>>>
>>>> -Farhad
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Sun, Nov 29, 2009 at 11:17 AM, shirazi <me@farhad.eu> wrote:
>>>>>>
>>>>>> I'm not sure i quite understand what you mean. Are you suggesting
that
>>>>>> in
>>>>>> order to have the throttling policy to kick in, I should make the
>>>>>> route
>>>>>> threaded/concurrent ?!
>>>>>
>>>>> You are using a file consumer which is using a single thread by
>>>>> default to consume files, and hence there will only be at most 1 in
>>>>> flight exchange in progress.
>>>>>
>>>>> Then you have a multicast where you send the file to 2 endpoints. the
>>>>> multicast is by default non parallel either. If you enable
>>>>> parallelProcessing() on the multicast you would then have 2 exchanges
>>>>> in flight, one for each of the multicasted destinations. However those
>>>>> destinations dont do much only write the file to another folder. So
>>>>> everything happens so fast.
>>>>>
>>>>> The dynamic throttler in this example dont make much sense as there
>>>>> are just not enough messages concurrently being routed. Instead you
>>>>> either have to
>>>>> - use more threads for processing concurrent files using threads (see
>>>>> the blog post)
>>>>> - create a lot of files so there are enough files to process
>>>>>
>>>>> Or try this example which is based on JMS where its more common to
>>>>> have a lot of messages being routed
>>>>> http://camel.apache.org/route-throttling-example.html
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> -Farhad
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> On Sun, Nov 29, 2009 at 9:32 AM, shirazi <me@farhad.eu>
wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Please have a look at the following route def.:
>>>>>>>>
>>>>>>>> RouteBuilder:
>>>>>>>> ==========
>>>>>>>>
>>>>>>>> fromF("file:///tmp/start").routeId("test.route.1").routePolicyRef("myPolicy").
>>>>>>>>        multicast().delay(1000).
>>>>>>>>                to("file:///tmp/proceed1").
>>>>>>>>                to("file:///tmp/proceed2");
>>>>>>>>
>>>>>>>>
>>>>>>>> from("file:///tmp/proceed2").routeId("test.route.2").routePolicyRef("myPolicy").delay(20000).
>>>>>>>>        to("file:///tmp/archive");
>>>>>>>>
>>>>>>>> applicationContext.xml
>>>>>>>> ==========
>>>>>>>>    <bean id="myPolicy"
>>>>>>>> class="org.apache.camel.impl.ThrottlingInflightRoutePolicy">
>>>>>>>>        <property name="scope" value="Context"/>
>>>>>>>>        <property name="maxInflightExchanges" value="3"/>
>>>>>>>>        <property name="resumePercentOfMax" value="40"/>
>>>>>>>>        <property name="loggingLevel" value="WARN"/>
>>>>>>>>    </bean>
>>>>>>>>
>>>>>>>> The throttling policy does not take effect, since files keep
coming
>>>>>>>> into
>>>>>>>> the
>>>>>>>> tmp/proceed dirs, regardless
>>>>>>>> the policy that says no more than 3 files at a time. Now
I am
>>>>>>>> clearly
>>>>>>>> doing
>>>>>>>> something wrong, but according
>>>>>>>> to the doc. what I have here should be enough.
>>>>>>>>
>>>>>>>
>>>>>>> No the route is fine. The file consumer is not concurrent, and
>>>>>>> neither
>>>>>>> is the multicast by default.
>>>>>>>
>>>>>>> Read this blog entry
>>>>>>> http://davsclaus.blogspot.com/2009/05/on-road-to-camel-20-concurrency-with.html
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Farhad S.
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/ThrottlingInflightRoutePolicy-does-not-take-effect-in-DSL-tp26560144p26560144.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/ThrottlingInflightRoutePolicy-does-not-take-effect-in-DSL-tp26560144p26560485.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/ThrottlingInflightRoutePolicy-does-not-take-effect-in-DSL-tp26560144p26562271.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>>
>
> --
> View this message in context: http://old.nabble.com/ThrottlingInflightRoutePolicy-does-not-take-effect-in-DSL-tp26560144p26562504.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



-- 
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Mime
View raw message