camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claus Ibsen <claus.ib...@gmail.com>
Subject Re: Performance and MessageSupport.getBody (1.6.0)
Date Wed, 04 Mar 2009 09:56:40 GMT
Hi Dan

Thank you very much for reporting and doing the detective work of
digging into the codebase and using a probe to detect the hot spot.

I am about to commit a fix that avoids the excessive overhead of the
NoTypeConverterException.

In a simple test sending 1000 messages to a route, I get about 2x
times the performance gain.
Going from about 9 sec to 4,5 sec.

This issue was caused by a StreamCacheInterceptor that was needed for
better integration with ServiceMix.
It will inspect the payloads for streams that are cache able and this
that is why its asking for the payload to be converted to a
StreamCache.

I added a rude unit test to the camel code to output the times
org.apache.camel.processor.RoutePerformanceTest

The ticket is:
https://issues.apache.org/activemq/browse/CAMEL-1417

You are welcome to try your test again with a new SNAPSHOT of Camel
2.0 or 1.6.1 when I get it committed.
We would be interested to hear your feedback and results.



On Wed, Mar 4, 2009 at 9:24 AM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
> Hi
>
> I am looking into this now. Its an issue introduced by the StreamCache
> eg:
> No type converter available to convert from type: java.lang.Integer to
> the required type: org.apache.camel.StreamCache with value 1
>
> Will create a ticket and get onto it
>
> On Tue, Mar 3, 2009 at 6:04 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>> Hi
>>
>> Did we see which type it tries to convert to? You have only said it
>> throws 10 exceptions but it should log which type it wanted to convert
>> it to.
>> That will also help.
>>
>>
>> On Tue, Mar 3, 2009 at 4:35 PM, Claus Ibsen <claus.ibsen@gmail.com> wrote:
>>> On Tue, Mar 3, 2009 at 4:19 PM, paquettd <dan.paquette@lmco.com> wrote:
>>>>
>>>> Not only avoiding the try..catch but just running through all the reflection
>>>> code over and over again can't be cheap either.
>>> It has a local cache so the lookup will be fast as its a map.
>>>
>>> But its the throws .. catch exception that is expensive if it does it
>>> 10 times for your test case.
>>> We gotta check into this. Maybe there is something obvious wrong.
>>>
>>> We can later improve it to not thrown the excpetion in certain
>>> situations where we dont care.
>>>
>>>
>>>>
>>>>
>>>> willem.jiang wrote:
>>>>>
>>>>> Maybe we could add some kind of type converter cache to avoid the  try
>>>>> ... catch work to improve the performance.
>>>>>
>>>>> Willem
>>>>>
>>>>> paquettd wrote:
>>>>>> Thank you for trying to help. I'm attaching my config file here.
As you
>>>>>> can
>>>>>> see this is a contrived example I made to test performance (I used
a
>>>>>> similar
>>>>>> example with some other products).
>>>>>>
>>>>>> Essentially what's happening is a String is being sent with the
>>>>>> ProducerTemplate send method; setting the type to InOut. The string
is
>>>>>> checked for lower case characters; and if it has any it goes to the
>>>>>> "capitalizer"; after that it goes to a reverser and truncator before
>>>>>> finally
>>>>>> a timestamp is put into the header (before I sent it I put a timestamp
as
>>>>>> well). I run this a few thousand times; using System.nanoTime for
those
>>>>>> timestamps (while not real accurate I used the same thing with other
COTS
>>>>>> and the relative differences are in time are larger than the clock
>>>>>> granularity)
>>>>>>
>>>>>> After I started seeing long processing times I used JProbe from Quest
>>>>>> Software to narrow down to the bottleneck we are talking about. JProbe
>>>>>> tells
>>>>>> me that Exception is thrown 10 times processing one of these messages....
>>>>>> so
>>>>>> when I send 1000 messages 10,000 of those exceptions are being thrown
and
>>>>>> it
>>>>>> takes a very long time for the 1000 messages to get through.
>>>>>>
>>>>>>
>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>
>>>>>> xmlns:camel="http://activemq.apache.org/camel/schema/spring"
>>>>>>      xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>
>>>>>> http://www.springframework.org/schema/beans/spring-beans.xsd
>>>>>>      http://activemq.apache.org/camel/schema/spring
>>>>>> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>>>>>>
>>>>>>      <bean id="reverser" class="camel.test.StringReverser"
/>
>>>>>>      <bean id="capitalizer" class="camel.test.StringCapitalizer"
/>
>>>>>>      <bean id="truncater" class="camel.test.StringTruncater">
>>>>>>              <property name="length" value="10" />
>>>>>>      </bean>
>>>>>>         <bean id="endTimestamper"
>>>>>> class="camel.test.transformers.HeaderTimestamper">
>>>>>>             <property name="headerKey" value="END_TIME"/>
>>>>>>         </bean>
>>>>>>
>>>>>>      <camelContext id="camel.test.camelcontext"
>>>>>>              xmlns="http://activemq.apache.org/camel/schema/spring">
>>>>>>              <template id="myCamelTemplate" defaultEndpoint="direct:test"
/>
>>>>>>
>>>>>>              <route>
>>>>>>                      <from uri="direct:test" />
>>>>>>                      <choice>
>>>>>>                              <when>
>>>>>>                                      <simple>${body}
regex '.*[a-x].*'</simple>
>>>>>>                                      <to uri="direct:capitalizeFirst"
/>
>>>>>>                              </when>
>>>>>>                              <otherwise>
>>>>>>                                      <to uri="direct:reverseAndTruncate"
/>
>>>>>>                              </otherwise>
>>>>>>                      </choice>
>>>>>>              </route>
>>>>>>
>>>>>>              <route>
>>>>>>                      <from uri="direct:capitalizeFirst"
/>
>>>>>>                      <bean ref="capitalizer" />
>>>>>>                      <to uri="direct:reverseAndTruncate"/>
>>>>>>              </route>
>>>>>>
>>>>>>              <route>
>>>>>>                      <from uri="direct:reverseAndTruncate"
/>
>>>>>>                      <bean ref="reverser" />
>>>>>>                      <bean ref="truncater" />
>>>>>>                      <to uri="bean:endTimestamper?method=timestamp"/>
>>>>>>              </route>
>>>>>>      </camelContext>
>>>>>>
>>>>>>      <bean id="testDriver" class="camel.test.TestDriver">
>>>>>>              <property name="producerTemplate" ref="myCamelTemplate"
/>
>>>>>>      </bean>
>>>>>> </beans>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>> On Tue, Mar 3, 2009 at 1:17 AM, Willem Jiang <willem.jiang@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> If you don't want the TypeConverter to get invovled , you
could use
>>>>>>>> MessageSupport.getBody() directly.
>>>>>>> Yes I am wondering if he has a .convertBodyTo in the route, so
we need
>>>>>>> to see this.
>>>>>>>
>>>>>>> Or some other endpoint/producer trying to get the body as a special
>>>>>>> type, eg InputStream etc.
>>>>>>>
>>>>>>> But the route would really help.
>>>>>>>
>>>>>>>
>>>>>>>> Willem
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Mar 3, 2009 at 1:43 AM, paquettd <dan.paquette@lmco.com>
wrote:
>>>>>>>>
>>>>>>>>> I'm not sure if it makes a difference but I'm not using
JMS anywhere.
>>>>>>>>> In
>>>>>>>>> fact
>>>>>>>>> in this test everything is using "direct".
>>>>>>>>>
>>>>>>>>> Is there something I can do in the Spring DSL to hint
to Camel that
>>>>>>>>> there
>>>>>>>>> is
>>>>>>>>> no conversion necessary?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>> On Mon, Mar 2, 2009 at 5:54 PM, paquettd <dan.paquette@lmco.com>
>>>>>>>>> wrote:
>>>>>>>>>>> I've been seeing some performance problems with
Camel 1.6.0 (I have
>>>>>>>>> not
>>>>>>>>>>> tried
>>>>>>>>>>> this with previous versions yet).
>>>>>>>>>>>
>>>>>>>>>>> My profiler is pointing the finger at MessageSupport.getBody,
>>>>>>>>>>> TypeConverter.convertTo, and DefaultTypeConverter.findTypeConverter
>>>>>>>>>>> specifically
>>>>>>>>>>>
>>>>>>>>>>> findTypeConverter is always throwing a
>>>>>>>>>>> NoTypeConversionAvailableException;
>>>>>>>>>>> which is then being caught and ignored in MessageSupport.getBody;
at
>>>>>>>>>>> which
>>>>>>>>>>> point processing continues successfully.
>>>>>>>>>> This should be normal in situations where you ask
the body to be a
>>>>>>>>>> specific type which cannot be converted to.
>>>>>>>>>> To help this can you show your route and what kind
of JMS messages
>>>>>>>>>> are
>>>>>>>>>> you sending.
>>>>>>>>>>
>>>>>>>>>> Camel is payload type agnostics (eg dont have to
be pure XML etc.) so
>>>>>>>>>> you can send whatever objects you like.
>>>>>>>>>> It has a rich type converter registry to be able
to convert seamless
>>>>>>>>>> between types.
>>>>>>>>>>
>>>>>>>>>> This registry is loaded on demand, so you should
make sure your start
>>>>>>>>>> profiling after Camel is "warm".
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> protected <T> T getBody(Class<T>
type, Object body) is the specific
>>>>>>>>>>> getBody
>>>>>>>>>>> in question.
>>>>>>>>>>>
>>>>>>>>>>> Is this exception an expected behavior? It's
weird how the catch
>>>>>>>>> block
>>>>>>>>>>> doesn't even log a warning. Should a converter
have been found? My
>>>>>>>>>>> message
>>>>>>>>>>> payload is just a java.lang.String.
>>>>>>>>>> In the old days it returned null, but that did not
work as the
>>>>>>>>>> payload
>>>>>>>>>> you were trying to convert could be null, so it was
a catch-22
>>>>>>>>>> situation.
>>>>>>>>>> So we added the exception.
>>>>>>>>>>
>>>>>>>>>> But if throwing exception is expensive we could maybe
add a has test
>>>>>>>>>> to avoid this exception being thrown and caught in
the
>>>>>>>>>> MessageSupport.
>>>>>>>>>>
>>>>>>>>>> The exception is also meant for end users so they
get a good
>>>>>>>>>> exception
>>>>>>>>>> detailing the problem if they try to convert something
into eg,
>>>>>>>>>> MyFooClass and its not possible to convert to it.
Instead of
>>>>>>>>>> returning
>>>>>>>>>> a null value as result.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I suspect I've done something wrong but I don't
know where to start
>>>>>>>>>>> looking.
>>>>>>>>>>> I'm concerned with this; as I'm comparing Camel
to some other
>>>>>>>>>>> message
>>>>>>>>>>> routing solutions. This is making Camel take
40 times longer than
>>>>>>>>>>> the
>>>>>>>>>>> competition and I want to make sure I do a fair
comparison.
>>>>>>>>>> We are currently rewamping the internal API in Camel
2.0 that leads
>>>>>>>>>> us
>>>>>>>>>> up to a point where we can do performance improvements
when Camel
>>>>>>>>>> routes exchanges.
>>>>>>>>>> Currently it does a bit of defensive copying when
it moves message
>>>>>>>>>> from node to node. The revamped API lets us do some
more clever stuff
>>>>>>>>>> there to improve the speed.
>>>>>>>>>>
>>>>>>>>>> So if you are testing, eg. JMS -> JMS and want
it to be really fast
>>>>>>>>>> then of course pure JMS to JMS is faster than eg
over Camel as its a
>>>>>>>>>> very flexible and transport/protocol agnostic framework.
But
>>>>>>>>>> performance improvements is on our roadmap in 2.1.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>>
>>>>>>>>> http://www.nabble.com/Performance-and-MessageSupport.getBody-%281.6.0%29-tp22291841p22291841.html
>>>>>>>>>>> Sent from the Camel - Users mailing list archive
at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>
>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://www.nabble.com/Performance-and-MessageSupport.getBody-%281.6.0%29-tp22291841p22292939.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context: http://www.nabble.com/Performance-and-MessageSupport.getBody-%281.6.0%29-tp22291841p22310649.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
>



-- 
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/

Mime
View raw message