camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Müller <christian.muel...@gmail.com>
Subject Re: Painfully slow AMQ consumer
Date Fri, 13 Jun 2014 17:03:04 GMT
Have a look at slide 16 in [1]. You can find the source code at [1].
As Claus wrote, getting the write lock to update the file - line by line -
is not really efficient. Aggregate some lines before your update the file
or use the stream component.
I could achieve to write more than 15.000 lines (140 bytes each) per
SECOND...

Hope this will help.

[1] http://www.slideshare.net/muellerc/apache-con-na-2013
[2] https://github.com/muellerc/apachecon-na-2013
[3] http://camel.apache.org/stream.html

Best,
Christian
-----------------

Software Integration Specialist

Apache Member
V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer
Apache Incubator PMC Member

https://www.linkedin.com/pub/christian-mueller/11/551/642


On Wed, May 21, 2014 at 6:06 AM, nw31304 <david.niedzielski@gmail.com>
wrote:

> I am attempting to do something really simple: read from an ActiveMQ queue
> and append the messages to a file.  Each message is roughly 100 bytes in
> length and the arrival rate is no more than 1000 messages per minutes.  No
> matter what I try, messages queue up on the broker because the Camel route
> does not process them quickly enough (i.e. append them to the file).  The
> disk I am writing to is SAN attached, an more than capable of these paltry
> requirements. I have ridiculous amounts of CPU at my disposal, an I/O
> utilization and queue time is minimal.  The machine load average on both
> the
> broker and the machine where I'm running camel is next to nothing, and none
> of the relevant machines are short of memory.  It doesn't matter if I run
> the broker on the same machine or on a different machine, the result is the
> same: a slowly growing queue.  If I tail the file,  literally see messages
> scrolling slowly enough to read them as they're written.  I'm using Camel
> 2.9.0, and ActiveMQ 5.5 (both client and broker).  I thought perhaps it was
> an issue with connection esablishment and teardown, but when I display the
> connections, it appears that the same set of connections are being used.
> Here's the relevant beans and routes:
>
>     <bean id="jmsConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
>           <property name="brokerURL" value="tcp://gsdb1:61616" />
>     </bean>
>
>     <bean id="pooledConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactory"
> init-method="start"
> destroy-method="stop">
>         <property name="maxConnections" value="20" />
>         <property name="connectionFactory" ref="jmsConnectionFactory" />
>     </bean>
>
>     <bean id="jmsConfig"
> class="org.apache.camel.component.jms.JmsConfiguration">
>         <property name="connectionFactory" ref="pooledConnectionFactory"/>
>         <property name="concurrentConsumers" value="5"/>
>     </bean>
>
>     <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
>         <property name="configuration" ref="jmsConfig"/>
>     </bean>
>
>     <camelContext id="camel1" xmlns="http://camel.apache.org/schema/spring
> ">
>
>         <jmxAgent id="agent" createConnector="true" registryPort="20023"/>
>
>         <route>
>             <from uri="activemq:gpsdata.gps.incoming.australia" />
>             <to
> uri="file://../archive-australia?fileName=gpsdata.log&amp;fileExist=Append"
> />
>     </camelContext>
>
>
> Can anyone lend a hand with this, or perhaps give me hand as to where to
> look?
>
> Thanks in advance.
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Painfully-slow-AMQ-consumer-tp5751382.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message