activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Strachan <>
Subject Re: [activemq-user] JMS - Streaming Data
Date Mon, 06 Mar 2006 10:11:31 GMT
On 3 Mar 2006, at 19:44, wrote:
> Hi all,
> I have a requirement where I need to collect streaming data from  
> all the exchanges and store them to database. I am planning on  
> using JMS (ActiveMQ) as a buffer in between.
> The thing is, my receiving application needs to be fast enough to  
> keep reading the stream from the exchanges, or else the exchange  
> servers start dropping messages since the client cannot keepup.
> Now what I am planning on doing is - read the messages from server,  
> translate them to java object, and keep firing them in a JMS topic.

So the first thing I'd suggest is to use async send with your  
publisher; then the thread thats reading from the stream of the  
exchange will not block to do a send(), rather an in-memory linked  
list will be used which then asynchronously streams the messages to  
the broker. This wi

> Then i will have a receiver application that subscribes as a  
> listener to the topic, and this listener application would do the  
> actual work of inserting records in the database. This way, I can  
> off-load the "buffering" task to JMS provider.
> The thing here is that database write operations are often slow,  
> and definately wont be able to keep up with the speed of incoming  
> stream. Is JMS (ActiveMQ) capable of buffering significanly large  
> amount of data? ( I dont mind putting lots and lots of RAM !) To  
> give a rough idea, the stream would mean atleast a few thousands of  
> messages per second ...

A few thousand messages per second on non-persistent topics is not  
that high a message rate. ActiveMQ can do > 20,000 message/second for  
one producer & consumer on a durable opteron box for example. YMMV

So ActiveMQ is already capable of buffering large amounts of messages  
in the JMS client so that there are thousands of messages available  
to process in RAM ASAP. Set a nice healthy prefetch size and you'll  
be fine

Another trick to boosting performance is to use JDBC batching;  
individual JDBC transactions are really slow - however if you are  
inserting 1000s of rows, use a JDBC batch and perform 1000 or 10,000  
inserts in a single transaction which is typically much faster.


View raw message