activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: starting a consumer on a queue with lots of messages
Date Tue, 28 Nov 2006 14:50:15 GMT
I would look at either of these - let us know how you get on!

On 27 Nov 2006, at 13:42, mark angrish 2 wrote:

>
> Also, like in the other thread you recommended, do you think i  
> should look at
> another storage mechanism other than derby (like a kaha store, or  
> postgres?)
>
>
>
> rajdavies wrote:
>>
>> Hi Mark,
>> answers inlined:
>>
>> On 27 Nov 2006, at 06:59, mark angrish 2 wrote:
>>
>>>
>>> Hi,
>>>
>>> I've had a search around but couldn't find anything that could help.
>>>
>>> I currently have around 2 million messages that are persisted in my
>>> queue.
>>> The situation I'm in is that the queue must hold onto 1 months (say
>>> 200Gb)
>>> worth of data before I can put consumers on to process the  
>>> backlog.  I
>>> currently only have the default config file setup.  My consumer is
>>> set up to
>>> use XA between activemq 4.0.2 and oracle 10g using atomikos
>>> transaction
>>> manager 3 integrated with spring 2.0, which allows me to use
>>> message driven
>>> pojos!  I am using the marathon fix (if you search the bug fix area
>>> you will
>>> find the patch) for activemq to allow it to work with XA.
>>>
>>> My questions:
>>>
>>> 1. My problem is that when i start active mq with the small 2
>>> million record
>>> scenario, it takes ages to start up.  Is there any way to overcome
>>> this poor
>>> startup time?
>>   There's new functionality that is in trunk at the moment, but is
>> due to be released in ActiveMQ 4.2 that solves this. In the current
>> release, all messages are loaded into the broker and a message
>> reference is then held in memory that has a pointer to the message in
>> the store. This can be optionally replaced by a paging mechanism that
>> pages messages into the broker when required.
>> However, there may be some work required to optimize some of the
>> queries for oracle - see http://www.nabble.com/Statements.java%2C-
>> etc.-Performance-tf2662372.html#a7425760
>>>
>>> 2. This is actually my big pain point at the moment.  When activemq
>>> does
>>> eventually start up with the 2 million messages, I can't use hermes
>>> to see
>>> the messages (it seems to take hours and still no result), and i
>>> can't see
>>> any processing happening on the xa consumer side to the database.
>>> When used
>>> normally say 10's or 100's of messages, so a steady stream of
>>> messages,
>>> everything processs to the database just fine.  I tried tinkering
>>> with the
>>> prefetch limit but that didn't seem to help either.  Something that
>>> could
>>> say "right, let the consumer take x amount of messages off the
>>> queue and
>>> process them with XA" would be awesome here.
>> Hopefully the paging mechanism will help here too.
>>
>>>
>>> 3. We have about 6 queues on 6 machines with 24 applications  
>>> acting as
>>> producers into them (so 4 producers going to each queue).  We have
>>> a SAN set
>>> up so that each queue can write to its own storage area on the san
>>> (i.e. 6
>>> activemq data directories on the SAN).  If i wanted to set up a
>>> store and
>>> forward kind of setup to 2 'central queues' where the consumers of
>>> the queue
>>> lie, do i just configure a network connector on the 6 queues to
>>> point to the
>>> central queue locations and get those central queues to read from
>>> all 6 SAN
>>> data directories?  Or do i actually need 6 consumer queues?  In
>>> either case
>>> is it ok for them to point to the SAN to read the message stores,
>>> considering the producers are writing and the consumers are  
>>> removing?
>>>
>>> Any help would be massively appreciated!
>>>
>> There isn't a problem with either 2 central queues or 6 consumer
>> queues - but I'd like to understand a little more about your setup -
>> do you really need store and forward, or could you just have a
>> centralized hub around your SAN ?
>>
>>> cheers,
>>>
>>> ::mark
>>> -- 
>>> View this message in context: http://www.nabble.com/starting-a-
>>> consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7555664
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>> cheers,
>>
>> rob
>>
>>
>
> -- 
> View this message in context: http://www.nabble.com/starting-a- 
> consumer-on-a-queue-with-lots-of-messages-tf2710121.html#a7560162
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


Mime
View raw message