qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: Qpid Java Broker with persistance supports only finite number of messages?
Date Thu, 05 Jan 2012 19:15:01 GMT
Yeah there are lots of interesting subtleties with this sort of thing :-)

I've got a bit of a mantra on my project "handle edge cases at the 
edges" so my "core" is very simple with federated C++ brokers run on 
boxes with lots of memory running nothing but C++ qpid brokers.

Which is why I've been looking into more "off the wall" approaches such 
as the one I mention using QMF queueThresholdExceeded.

You make some interesting points wrt "

Flowing
to disk may make sense here as long as the rate at which messages can flow
through the disk is higher than the rate at which they are entering the
broker (and actually high enough so that the backlog can quickly be
reduced)

"

Like I say it's subtle. In my case I can tolerate  moderate loss, so the 
WAN failure is and edge case for me. Even if the disk rate isn't higher 
than the input rate I'll still save *something*. In my case the latest 
messages are more valuable than older ones so I'm likely to "replay" 
using a LIFO model too.


On 05/01/12 18:57, Rob Godfrey wrote:
> On 5 January 2012 19:30, Fraser Adams<fraser.adams@blueyonder.co.uk>  wrote:
>
>> Just to jump in on this thread.
>>
>> Re "
>>
>>
>> but my opinion
>> is that if you have "millions of messages" then a Message Broker is the
>> wrong solution to your problem - you want a Database.
>>
>> "
>> I can't say I agree with Rob's assertion here!!
>>
>> Well maybe that's a reasonable comment if the *intention* is to have
>> millions of messages hanging around, but what if it's due to an unfortunate
>> circumstance......
>>
>>
> Indeed - that is what I meant :-)  Planning for failure scenarios should
> always be carefully considered... As Robbie mentioned we have had people
> build up millions of messages on persistent queues... and if you use the
> 64-bit JVM and put large amounts of RAM in your box, then the physical
> limitations of maximum queue size can be made acceptably large... However I
> would always caution against designing a system where you expect to hold
> this number of *persistent* messages as part of your regular process -
> especially using the Java Broker as it is not designed for this sort of use
> case.
>
> The case you describe is somewhat different in that you are looking at
> transient messages (where one would expect the rate of
> production/consumption to be higher), and you want the broker to be capable
> of buffering for N minutes while temporary network outages occur.  Flowing
> to disk may make sense here as long as the rate at which messages can flow
> through the disk is higher than the rate at which they are entering the
> broker (and actually high enough so that the backlog can quickly be
> reduced).  This is where the flow-to-disk work that I hope to be starting
> soon on the Java Broker would play.
>
> -- Rob
>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message