qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "GS.Chandra N" <gs.chandra...@gmail.com>
Subject Subscriptions slows down broker (was Memory pile up on broker)
Date Fri, 13 Mar 2009 19:45:38 GMT
Gordon, Tedd

After the patches to the headersexchange.cpp was applied, i can now clearly
see the incoming rates with the help of the tool Tedd sent.

The rates are at about 90-100 odd messages per sec (a drop from 5200 odd
without subscriptions).

Sort of matches my earlier observations using qpid-tool. What next?

Do you think there might be any way to add the headers exchange performance
improvements to your list of priority fixes? I'm not on the dev forums so i
'm not sure if the issue is communicated out there.

I will try to test the multiple subscriptions XML exchange next to if that
improves matters in any respect.

Thanks for the great support in fully un-reavelling the issue.

Rgds
gs

ps : I can also still reproduce the memory pile up by blocking the broker
using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i
can avoid by not letting qpid-tool stick around and increasing the mgmt-pub
interval to 10 or more.

On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs.chandran.n@gmail.com>wrote:

> Hi,
>
> I found that if i kept the qpid-tool open long enough all my clients and
> publishers would throw exceptions and come out.
>
> Earlier when i was performing my tests i had 3 putty windows opened to all
> the 3 servers and it seems as if, i was keeping qpid-tool opened just long
> enough to cause build up but closing it to run top and so on a so forth such
> that the memory piled up but did not cause timeouts at the client either.
>
> Observations
>
> 1.When there are subscriptions and a qpid-tool around messages seem to pile
> up. Why is this so? *
> *
> 2. When there are no subscribers and no qpid-tool the publishing processes
> cause the CPu to go to 90% at the publishing box. But when i started up
> subscribers, the publishing box CPU fell to 0% and all the python processes
> piled up memory.
>
> Concern : I'm not able to find out here if the publishers are still able to
> send out messages at the speed that it should when subscriptions are around.
> Or perhaps the broker is not able to pull enough messages - iam not sure
> which of this is happeneing. Probably latter? How do i tell?
>
> If i use qpid-tool to connect to the broker at this point, every time i hit
> show exchange i get the same stats with the same time stamp. it looks like
> the broker is too busy trying to match subscriptions even to refresh the
> stats.
>
> 3.Concern : Once the subscriptions are made, the CPU at the broker box* *(high
> end dual core XEON with ht) goes to 90%.  Even at this level, i'm not sure
> the broker is able to match and discard all the messages fast enough (due to
> 2nd observation). Or how do i tell?
>
>
> Thanks
> gs
>
> ps : Is there a sure shot way to find out the message rates ? (I currently
> use qpid-tool show exchange command to find the no of msgRecieves and divide
> by time-dfference from 2 times the command is run)
>
>
> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <gs.chandran.n@gmail.com>wrote:
>
>> I'm able to reproduce the issue, when i keep qpid-tool open all the time
>> with the mgmt switches.
>>
>> However i'm not able to do so if qpid-tool is not open (did not try with
>> qpid-tool and no mgmt switches).
>>
>> Thanks
>> gs
>>
>>
>> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gsim@redhat.com> wrote:
>>
>>> GS.Chandra N wrote:
>>>
>>>> ps : Please find attached the scripts that create the issue
>>>>
>>>
>>> I'm afraid I wasn't able to recreate the issue with your tests. Memory
>>> stayed reasonable and constant as the test was running (5 servers, 5
>>> clients).
>>>
>>> I'm baffled at this point as to what it is you are seeing.
>>>
>>>  broker - runs on a dedicated box
>>>> qpidd -d --port=5672  --mgmt-enable=yes  --mgmt-pub-interval=1 --auth=no
>>>> --log-source=yes --log-to-file=/tmp/somename
>>>>
>>>
>>> Do you have qpid-tool or anything else running during the test? Does
>>> turning management off have any impact on the issue?
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project:      http://qpid.apache.org
>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message