qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: C++ broker memory leak in federated set-up???
Date Thu, 01 Mar 2012 19:30:58 GMT
Re

"Your sure there is no backed up queue anywhere?"

Pretty sure, it's a C++ producer sending messages to amq.match to a 
broker on the same host, that broker has a single 1GB queue for the 
queue route federated to the destination broker. The queue for the queue 
route has been set up with ring policy e.g. circular queue.

At some stage the queue could well have filled, but it's a circular 
queue so I wouldn't expect to see qpidd consume more than a few GB 
certainly not 35GB

So what we're seeing is

crappy network = broker memory creeps up until the system eventually 
dies under the weight of swapping
decent network = broker memory behaves as one would expect

We're seeing connection closed messages in the logs when the network is 
behaving poorly.

I'd agree that It does seem weird that simple reconnections should 
appear to cause such a large growth but it does appear to be a 
consistent pattern and I can't think of any other explanation.

The queue is a 1GB circular non-persistent queue bound to amq.match via 
a couple of fairly basic headers.


I did wonder if there was some headers exchange weirdness going on as 
that seems less well supported than the other message patterns, we're 
using queue routes because we want to be able to have a non-default size 
and ring policy, but with 0.8 we had no other choice anyway as there was 
a bug with dynamic routes (I know you fixed that in 0.12) and qpid-route 
doesn't support static routes using amg.match so I'm slightly suspicious 
that we might see something unusual as I think we may use the headers 
exchange more than most.

I'm not convinced the Jira you mentioned is related as we're definitely 
supplying the correct federation information, in our case the network 
problem seems to be causing the connection to drop periodically out and 
auto-reconnect.

Frase

On 01/03/12 15:04, Gordon Sim wrote:
> On 02/29/2012 07:07 PM, Fraser Adams wrote:
>> Hi All,
>> I think that we may have stumbled across a potential memory/resource 
>> leak.
>>
>> We have one particular set up where we have a C++ producer client (using
>> qpid::client - don't ask, it's a long story.....) this writes to a 0.8
>> broker hosted on the same server. That broker is then federated via a
>> queue route to amq.match on another (0.8) broker. The queue route is a
>> source route set up via qpid-route -s
>>
>> We've been having all sorts of fun and games with respect to
>> performance, which we've narrowed down to some dodgy networking.
>>
>> However one of the other effects that we've noticed is that the broker
>> co-located with the producer client eats memory. The queue for the queue
>> route is 1GB but qpidd eventually grows to ~35GB and sends the whole set
>> up into swap.
>>
>> So with respect to the network problem we're suspecting a dodgy switch
>> somewhere, what is interesting is that when we checked with ethtool the
>> NIC was reporting half duplex had been negotiated - ouch!!! hence why we
>> suspect a dodgy switch somewhere.
>>
>> Now when the NIC was explicitly set to 100 base/T full duplex our
>> performance rocketed and the broker on the producer system appears
>> (touch wood) to have stable memory performance.
>>
>> What I'm suspecting is that the dodgy network link has been causing
>> connection drop-outs and the broker is automatically reconnecting (logs
>> are confirming this) and I'm thinking that there is a resource leak
>> somewhere during the reconnection process.
>
> https://issues.apache.org/jira/browse/QPID-3447 perhaps? Though I 
> wouldn't have expected that to cause such a large growth in memory.
>
> Your sure there is no backed up queue anywhere?
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message