activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Strachan <james.strac...@gmail.com>
Subject Re: [activemq-user] JMS - Streaming Data
Date Mon, 06 Mar 2006 10:11:31 GMT
On 3 Mar 2006, at 19:44, shitalbpatel@optonline.net wrote:
> Hi all,
>
> I have a requirement where I need to collect streaming data from  
> all the exchanges and store them to database. I am planning on  
> using JMS (ActiveMQ) as a buffer in between.
>
> The thing is, my receiving application needs to be fast enough to  
> keep reading the stream from the exchanges, or else the exchange  
> servers start dropping messages since the client cannot keepup.
> Now what I am planning on doing is - read the messages from server,  
> translate them to java object, and keep firing them in a JMS topic.

So the first thing I'd suggest is to use async send with your  
publisher; then the thread thats reading from the stream of the  
exchange will not block to do a send(), rather an in-memory linked  
list will be used which then asynchronously streams the messages to  
the broker. This wi


> Then i will have a receiver application that subscribes as a  
> listener to the topic, and this listener application would do the  
> actual work of inserting records in the database. This way, I can  
> off-load the "buffering" task to JMS provider.
>
> The thing here is that database write operations are often slow,  
> and definately wont be able to keep up with the speed of incoming  
> stream. Is JMS (ActiveMQ) capable of buffering significanly large  
> amount of data? ( I dont mind putting lots and lots of RAM !) To  
> give a rough idea, the stream would mean atleast a few thousands of  
> messages per second ...

A few thousand messages per second on non-persistent topics is not  
that high a message rate. ActiveMQ can do > 20,000 message/second for  
one producer & consumer on a durable opteron box for example. YMMV
http://activemq.org/Performance

So ActiveMQ is already capable of buffering large amounts of messages  
in the JMS client so that there are thousands of messages available  
to process in RAM ASAP. Set a nice healthy prefetch size and you'll  
be fine
http://docs.codehaus.org/display/ACTIVEMQ/What+is+the+prefetch+limit+for

Another trick to boosting performance is to use JDBC batching;  
individual JDBC transactions are really slow - however if you are  
inserting 1000s of rows, use a JDBC batch and perform 1000 or 10,000  
inserts in a single transaction which is typically much faster.

James
-------
http://radio.weblogs.com/0112098/


Mime
View raw message