qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Henry <whe...@redhat.com>
Subject Re: C++ broker memory leak in federated set-up???
Date Mon, 12 Mar 2012 23:52:50 GMT


----- Original Message -----
> On 03/09/2012 04:10 PM, Fraser Adams wrote:
> > On 09/03/12 14:22, Gordon Sim wrote:
> >>
> >> Can you (or have you) tracked the queue depth, connection and
> >> session
> >> stats for the broker exhibiting the problem? Anything you can
> >> think of
> >> that might correlate with the rate of growth (e.g. does it look
> >> like
> >> its per message)?
> >>
> > Unfortunately nothing looks especially anomalous. As I mentioned
> > earlier
> > in this thread we did have what I suspect was a dodgy network
> > switch
> > which caused the producer host NIC to negotiate half-duplex. That
> > hosed
> > network performance and caused things to back up on the (circular)
> > queue
> > and was also causing regular reconnections (not mega-fast ~every 10
> > minutes) so initially I wondered if that was where the problem may
> > lie,
> > however since we sorted the network problem it has been running as
> > fast
> > as I'd expect with nothing obvious in the logs (no reconnections
> > now)
> > and a queue that now stays pretty close to empty.
> >
> >>> As I say below we've got a 0.8 qpid::client producer delivering
> >>> to
> >>> amq.match on a broker co-located on the same host which is
> >>> federated to
> >>> another 0.8 broker (all brokers are c++) via a source queue
> >>> route.
> >>
> >> That is all that's happening on the problem broker?
> >>
> > Yeah the problem broker doesn't have much going on at all. It has a
> > single queue, the producer is writing to amq.match and there are a
> > couple of matches in the x-match-all binding. The queue is sized to
> > 1G
> > the server has 24GB. The queue is a basic in-memory ring queue so
> > no
> > persistence or anything going on.
> >
> > We are using source routes set up with qpid-config -s
> >
> > That has me thinking I guess that I should standard routes just to
> > rule
> > out source routes as a potential problem. I don't suppose you are
> > aware
> > of any potential issues with source routes (he says clutching at
> > straws....)
> 
> Definitely worth trying as that is the less travelled path and its
> always possible there is something we haven't seen before. There is
> nothing I'm aware of however.
> 
> Are you using acknowledgements on your federated bridge?

That's what I"m wondering.  You mention one exchange and one queue but 
with federation involved there is more going on.  Which queue is the
1GB ring queue? I assume the final queue in the federation. Therefore
I too wonder if the origin broker filling up because it's not getting
acks that the messages have been consumed. 

??

Can you check both brokers for exchange and queue stats?

William  

> 
> > The producer is a slightly unusual fairly real time thing that
> > mmaps
> > it's main chunk of memory and uses an internal memory pool to
> > optimise
> > multi-threaded memory access but I believe that it is only grabbing
> > ~8GB
> >> rpm -qv <name> will give you the versioned name of the rpms which
> >> may
> >> shed some light...
> >>
> > Is there a list of libraries that qpid has dependencies on
> > available
> > anywhere so I can compare the info I get from the above command
> > with
> > what *should* be in place on the host?
> 
> There is no formal list. Many (most?) of them are optional also, so
> it
> depends what you compiled with. Boost is the key dependency and you
> do
> want to make sure you use the same version of boost that you compiled
> against. Not doing so causes problems, though usually more pronounced
> than this so I don't suggest that is your issue.
> 
> What versions of the qpid rpms and boost are you using on the two
> boxes?
> 
> >> Nothing that should cause leaks... are you running any RHEL6?
> > I *think* it might be RHEL5, unfortunately I can't check at the
> > moment
> > as I'm mailing from home. Why did you ask about RHEL6, are there
> > any
> > issues with that?
> 
> There was an issue with the memory allocation strategy in RHEL6,
> where a
> the per-thread pools of memory didn't work well in the case of a
> thread
> that always worked on producing (hence allocating) and another that
> did
> all the consuming (hence freeing).
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message