activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manuel Teira Paz <mte...@tid.es>
Subject Re: Database access and prefetch considerations
Date Thu, 06 Nov 2008 17:43:40 GMT
Manuel Teira Paz escribió:
> Thanks a lot, James, for your fast answer. Taking advantage of your
> kindness, I will like to elaborate on some of the topics. Please, find
> my questions below:
>   
Hi. Bumping this message, to see if anyone skilled in activemq insides 
could answer my questions.

Best regards.


>
> James Strachan escribió:
>   
>> 2008/10/30 Manuel Teira Paz <mteira@tid.es>:
>>
>>     
>>> Hi, activemq guys.
>>>
>>>
>>> We are suffering some bad service times on a production system embedding
>>> ActiveMQ as message broker. We are not sure whether the problem is the
>>> broker itself, but I would like to get some insights in the broker
>>> internals, to be able to understand the situation better.
>>>
>>> We are using an activemq 4.1 release (we are not allowed to upgrade it) ,
>>> with pure JDBC persistence. We are using only queues and persistent
>>> messages.
>>>
>>> The first concern is about database access. Even when the number of messages
>>> in queues are high, we are observing no growth in the number of active
>>> database connections. Furthermore, it seems that only a thread per
>>> connection is responsible of fetching and deleting messages in the
>>> persistence.
>>>
>>>       
>> Yes thats how it works.
>>
>>
>>
>>     
>>> Given the fact that we have 30 consumers on a given queue,
>>> sharing the same connection, does this mean that a single thread is managing
>>> all the persistence needs? If true, wouldn't it be a potential bottleneck,
>>> avoiding us to improve the performance adding more consumers?
>>>
>>>       
>> ActiveMQ is tuned for loads of concurrent connections. If you only
>> really have one connection, just don't share your connection across
>> each producer/consumer - or use async dispatch
>>
>>
>>     
> Actually, we are not using a single connection for the whole system, but
> a single connection for every one of our components, handling message
> from a given queue. So, what is your recommendation, to use a single
> connection for every one of our 30 consumers, some ratio between
> consumers and connections?
>
> About the async dispatch possibility. What do actually involves,
> performance and reliably wise? I mean, how the database workload is
> delivered using async dispatch? Or, for example, should we be aware of
> asynchronous exceptions if we switch to that mode?
>   
>>> The second question is about the consumer prefetch. We did disable it in the
>>> past, trying to avoid situations where messages got stuck in queues, due to
>>> slow consumption of previous messages, producing unexpected and apparently
>>> random delays in the consumption of some messages.
>>>
>>> We wonder if prefetch could in some way lighten the load on the database
>>> side. We guess that perhaps messages prefetched for a given queue are kept
>>> in memory, avoiding the need to read them from the database. On the other
>>> side, does prefetch activate a particular revover strategy for database
>>> message restoring, or are they fetched in a one by one basis?
>>>
>>>       
>> They are fetched one by one - prefetch only affects how quickly they
>> are streamed to a consumer
>>
>>
>>
>>     
>>> This also triggers the question of mesage caching. Even when prefetch is
>>> disabled, is there any strategy, perhaps configurable, to cache messages in
>>> memory?
>>>
>>>       
>> Not in 4.x though there are improvements in 5.x
>>
>>     
> Does that mean that there isn't any cache feature in 4.x or that it's
> not configurable?
> Could you please give more details about those improvements in 5.x?
>
>   
>> BTW pure JDBC is always gonna be kinda slow - its why we developed AMQ
>> Store which is often 10X to 100X faster
>> http://activemq.apache.org/amq-message-store.html
>>
>>
>>     
> The problem here is that we need the system to work in a master-slave
> configuration. I think that having some kind of short term fast
> persistence solution (as I guess amq store is) is not an option, since
> on a machine failure, database information is not guaranteed to be
> updated. Or am I missing something?
>
>
> Thanks a lot. Looking forward for your wise answers.
>
> Best regards.
>
>
>   
>> --
>> James
>> -------
>> http://macstrac.blogspot.com/
>>
>> Open Source Integration
>> http://fusesource.com/
>>
>>     
>
>   


Mime
View raw message