activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mark angrish 2 <mark.angr...@gmail.com>
Subject Re: starting a consumer on a queue with lots of messages
Date Mon, 27 Nov 2006 13:42:41 GMT

Also, like in the other thread you recommended, do you think i should look at
another storage mechanism other than derby (like a kaha store, or postgres?)



rajdavies wrote:
> 
> Hi Mark,
> answers inlined:
> 
> On 27 Nov 2006, at 06:59, mark angrish 2 wrote:
> 
>>
>> Hi,
>>
>> I've had a search around but couldn't find anything that could help.
>>
>> I currently have around 2 million messages that are persisted in my  
>> queue.
>> The situation I'm in is that the queue must hold onto 1 months (say  
>> 200Gb)
>> worth of data before I can put consumers on to process the backlog.  I
>> currently only have the default config file setup.  My consumer is  
>> set up to
>> use XA between activemq 4.0.2 and oracle 10g using atomikos  
>> transaction
>> manager 3 integrated with spring 2.0, which allows me to use  
>> message driven
>> pojos!  I am using the marathon fix (if you search the bug fix area  
>> you will
>> find the patch) for activemq to allow it to work with XA.
>>
>> My questions:
>>
>> 1. My problem is that when i start active mq with the small 2  
>> million record
>> scenario, it takes ages to start up.  Is there any way to overcome  
>> this poor
>> startup time?
>   There's new functionality that is in trunk at the moment, but is  
> due to be released in ActiveMQ 4.2 that solves this. In the current  
> release, all messages are loaded into the broker and a message  
> reference is then held in memory that has a pointer to the message in  
> the store. This can be optionally replaced by a paging mechanism that  
> pages messages into the broker when required.
> However, there may be some work required to optimize some of the  
> queries for oracle - see http://www.nabble.com/Statements.java%2C- 
> etc.-Performance-tf2662372.html#a7425760
>>
>> 2. This is actually my big pain point at the moment.  When activemq  
>> does
>> eventually start up with the 2 million messages, I can't use hermes  
>> to see
>> the messages (it seems to take hours and still no result), and i  
>> can't see
>> any processing happening on the xa consumer side to the database.   
>> When used
>> normally say 10's or 100's of messages, so a steady stream of  
>> messages,
>> everything processs to the database just fine.  I tried tinkering  
>> with the
>> prefetch limit but that didn't seem to help either.  Something that  
>> could
>> say "right, let the consumer take x amount of messages off the  
>> queue and
>> process them with XA" would be awesome here.
> Hopefully the paging mechanism will help here too.
> 
>>
>> 3. We have about 6 queues on 6 machines with 24 applications acting as
>> producers into them (so 4 producers going to each queue).  We have  
>> a SAN set
>> up so that each queue can write to its own storage area on the san  
>> (i.e. 6
>> activemq data directories on the SAN).  If i wanted to set up a  
>> store and
>> forward kind of setup to 2 'central queues' where the consumers of  
>> the queue
>> lie, do i just configure a network connector on the 6 queues to  
>> point to the
>> central queue locations and get those central queues to read from  
>> all 6 SAN
>> data directories?  Or do i actually need 6 consumer queues?  In  
>> either case
>> is it ok for them to point to the SAN to read the message stores,
>> considering the producers are writing and the consumers are removing?
>>
>> Any help would be massively appreciated!
>>
> There isn't a problem with either 2 central queues or 6 consumer  
> queues - but I'd like to understand a little more about your setup -  
> do you really need store and forward, or could you just have a  
> centralized hub around your SAN ?
> 
>> cheers,
>>
>> ::mark
>> -- 
>> View this message in context: http://www.nabble.com/starting-a- 
>> consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7555664
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
> 
> cheers,
> 
> rob
> 
> 

-- 
View this message in context: http://www.nabble.com/starting-a-consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7560162
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message