activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: starting a consumer on a queue with lots of messages
Date Mon, 27 Nov 2006 08:23:18 GMT
Hi Mark,
answers inlined:

On 27 Nov 2006, at 06:59, mark angrish 2 wrote:

>
> Hi,
>
> I've had a search around but couldn't find anything that could help.
>
> I currently have around 2 million messages that are persisted in my  
> queue.
> The situation I'm in is that the queue must hold onto 1 months (say  
> 200Gb)
> worth of data before I can put consumers on to process the backlog.  I
> currently only have the default config file setup.  My consumer is  
> set up to
> use XA between activemq 4.0.2 and oracle 10g using atomikos  
> transaction
> manager 3 integrated with spring 2.0, which allows me to use  
> message driven
> pojos!  I am using the marathon fix (if you search the bug fix area  
> you will
> find the patch) for activemq to allow it to work with XA.
>
> My questions:
>
> 1. My problem is that when i start active mq with the small 2  
> million record
> scenario, it takes ages to start up.  Is there any way to overcome  
> this poor
> startup time?
  There's new functionality that is in trunk at the moment, but is  
due to be released in ActiveMQ 4.2 that solves this. In the current  
release, all messages are loaded into the broker and a message  
reference is then held in memory that has a pointer to the message in  
the store. This can be optionally replaced by a paging mechanism that  
pages messages into the broker when required.
However, there may be some work required to optimize some of the  
queries for oracle - see http://www.nabble.com/Statements.java%2C- 
etc.-Performance-tf2662372.html#a7425760
>
> 2. This is actually my big pain point at the moment.  When activemq  
> does
> eventually start up with the 2 million messages, I can't use hermes  
> to see
> the messages (it seems to take hours and still no result), and i  
> can't see
> any processing happening on the xa consumer side to the database.   
> When used
> normally say 10's or 100's of messages, so a steady stream of  
> messages,
> everything processs to the database just fine.  I tried tinkering  
> with the
> prefetch limit but that didn't seem to help either.  Something that  
> could
> say "right, let the consumer take x amount of messages off the  
> queue and
> process them with XA" would be awesome here.
Hopefully the paging mechanism will help here too.

>
> 3. We have about 6 queues on 6 machines with 24 applications acting as
> producers into them (so 4 producers going to each queue).  We have  
> a SAN set
> up so that each queue can write to its own storage area on the san  
> (i.e. 6
> activemq data directories on the SAN).  If i wanted to set up a  
> store and
> forward kind of setup to 2 'central queues' where the consumers of  
> the queue
> lie, do i just configure a network connector on the 6 queues to  
> point to the
> central queue locations and get those central queues to read from  
> all 6 SAN
> data directories?  Or do i actually need 6 consumer queues?  In  
> either case
> is it ok for them to point to the SAN to read the message stores,
> considering the producers are writing and the consumers are removing?
>
> Any help would be massively appreciated!
>
There isn't a problem with either 2 central queues or 6 consumer  
queues - but I'd like to understand a little more about your setup -  
do you really need store and forward, or could you just have a  
centralized hub around your SAN ?

> cheers,
>
> ::mark
> -- 
> View this message in context: http://www.nabble.com/starting-a- 
> consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7555664
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

cheers,

rob

Mime
View raw message