qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Praveen M <lefthandma...@gmail.com>
Subject Re: Qpid Java Broker with persistance supports only finite number of messages?
Date Wed, 04 Jan 2012 21:09:02 GMT
Thanks for the explanation Robbie.

On Wed, Jan 4, 2012 at 1:12 PM, Robbie Gemmell <robbie.gemmell@gmail.com>wrote:

> Hi Praveen,
>
> I can only really guess to any design decision on that front as it
> would have been before my time with the project, but I'd say its
> likely just that way because theres never been a strong need / use
> case that actually required doing anything else. For example, with
> most of the users I liase with the data they are using has at least
> some degree of time sensitivity to it and having anywhere near that
> volume of persistent data in the broker would represent some sort of
> ongoing period of catastrophic failure in their application. I can
> only really think of one group who make it into multi-million message
> backlogs at all, and that usually includes having knowingly published
> things which noone will ever consume.
>
> For a FIFO queue you are correct it would 'just' need to load in more
> as required. Things get trickier when dealing with some of the other
> queue types however, such as as LVQ/conflation and the recently added
> Sorted queue types. Making the broker able to hold partial segments of
> the queue in memory is something we have discussed doing in the past
> for other reasons, but message volume hasnt really been a significant
> factor in those considerations until now. I will take note of it for
> any future work we do in that area though.
>
> Robbie
>
> On 1 January 2012 17:46, Praveen M <lefthandmagic@gmail.com> wrote:
> > Hi,
> >
> > I was digging in the code base and was trying to understand how the
> broker
> > is implemented.
> > I see that for each message enqueued there are certain objects kept in
> > memory one for each message.
> >
> > example: MessageTransferReference, SimpleQueueEntryImpl etc.
> >
> > I tried computing the memory footprint of each individual message and it
> > amounts about 320 bytes/message.
> > I see that because of the footprint of each message,  if i'm limited to
> 4GB
> > of memory, then I am limited to only about 13 million messages in the
> > system at one point.
> >
> > Since I'm using a persistent store I'd have expected to go over 13
> million
> > messages and be limited by disk store rather than physical memory, but
> > I realized this isn't the case.
> >
> > I am curious as to what were the driving points for this design decision
> to
> > keep a reference to every message in memory. I'd have expected in a FIFO
> > queue you just need a subset of messages in memory and can  pull in
> > messages on demand rather than maintain reference to every message in
> > memory.
> >
> > Can someone please explain as to the reasons for this design? Also, was
> it
> > assumed that we'd never flood the queues over 13 million messages at one
> > time. Was there a bound
> > decided upon?
> >
> > Thank you,
> > Praveen
> >
> >
> >
> > --
> > -Praveen
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>
>


-- 
-Praveen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message