qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Trieloff <cctriel...@redhat.com>
Subject Re: Subscriptions slows down broker (was Memory pile up on broker)
Date Mon, 16 Mar 2009 20:53:33 GMT

attached, works with linux kernel with private-futex support (RHEL5.3 or 
F10 etc). If you run
an older kernel without private-futexes shout and I will give you that 
script.

Carl.


GS.Chandra N wrote:
> That ought to be interesting. I shall do the profiling and perhaps take a
> look at the code too. Please do send me the script.
>
> Thanks
> gs
>
> On Mon, Mar 16, 2009 at 6:09 AM, Carl Trieloff <cctrieloff@redhat.com>wrote:
>
>   
>> I don't believe any profiling has been done on the headers exchange. If you
>> are on Linux, run oprofile on it
>> and either create a patch to correct the hot spot or mail the profile to
>> the list.
>>
>> if needed I can also give mail you a stap (system tap) script for lock
>> analysis for the headers exchange.
>> regards,
>> Carl.
>>
>>
>>
>> GS.Chandra N wrote:
>>
>> Gordon, Tedd
>>
>> After the patches to the headersexchange.cpp was applied, i can now clearly
>> see the incoming rates with the help of the tool Tedd sent.
>>
>> The rates are at about 90-100 odd messages per sec (a drop from 5200 odd
>> without subscriptions).
>>
>> Sort of matches my earlier observations using qpid-tool. What next?
>>
>> Do you think there might be any way to add the headers exchange performance
>> improvements to your list of priority fixes? I'm not on the dev forums so i
>> 'm not sure if the issue is communicated out there.
>>
>> I will try to test the multiple subscriptions XML exchange next to if that
>> improves matters in any respect.
>>
>> Thanks for the great support in fully un-reavelling the issue.
>>
>> Rgds
>> gs
>>
>> ps : I can also still reproduce the memory pile up by blocking the broker
>> using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i
>> can avoid by not letting qpid-tool stick around and increasing the mgmt-pub
>> interval to 10 or more.
>>
>> On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs.chandran.n@gmail.com>wrote:
>>
>>     
>>> Hi,
>>>
>>> I found that if i kept the qpid-tool open long enough all my clients and
>>> publishers would throw exceptions and come out.
>>>
>>> Earlier when i was performing my tests i had 3 putty windows opened to all
>>> the 3 servers and it seems as if, i was keeping qpid-tool opened just long
>>> enough to cause build up but closing it to run top and so on a so forth such
>>> that the memory piled up but did not cause timeouts at the client either.
>>>
>>> Observations
>>>
>>> 1.When there are subscriptions and a qpid-tool around messages seem to
>>> pile up. Why is this so? *
>>> *
>>> 2. When there are no subscribers and no qpid-tool the publishing processes
>>> cause the CPu to go to 90% at the publishing box. But when i started up
>>> subscribers, the publishing box CPU fell to 0% and all the python processes
>>> piled up memory.
>>>
>>> Concern : I'm not able to find out here if the publishers are still able
>>> to send out messages at the speed that it should when subscriptions are
>>> around. Or perhaps the broker is not able to pull enough messages - iam not
>>> sure which of this is happeneing. Probably latter? How do i tell?
>>>
>>> If i use qpid-tool to connect to the broker at this point, every time i
>>> hit show exchange i get the same stats with the same time stamp. it looks
>>> like the broker is too busy trying to match subscriptions even to refresh
>>> the stats.
>>>
>>> 3.Concern : Once the subscriptions are made, the CPU at the broker box* *(high
>>> end dual core XEON with ht) goes to 90%.  Even at this level, i'm not sure
>>> the broker is able to match and discard all the messages fast enough (due to
>>> 2nd observation). Or how do i tell?
>>>
>>>
>>> Thanks
>>> gs
>>>
>>> ps : Is there a sure shot way to find out the message rates ? (I currently
>>> use qpid-tool show exchange command to find the no of msgRecieves and divide
>>> by time-dfference from 2 times the command is run)
>>>
>>> On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <gs.chandran.n@gmail.com>wrote:
>>>
>>>       
>>>> I'm able to reproduce the issue, when i keep qpid-tool open all the time
>>>> with the mgmt switches.
>>>>
>>>> However i'm not able to do so if qpid-tool is not open (did not try with
>>>> qpid-tool and no mgmt switches).
>>>>
>>>> Thanks
>>>> gs
>>>>
>>>> On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gsim@redhat.com> wrote:
>>>>
>>>>         
>>>>> GS.Chandra N wrote:
>>>>>
>>>>>           
>>>>>> ps : Please find attached the scripts that create the issue
>>>>>>
>>>>>>             
>>>>> I'm afraid I wasn't able to recreate the issue with your tests. Memory
>>>>> stayed reasonable and constant as the test was running (5 servers, 5
>>>>> clients).
>>>>>
>>>>> I'm baffled at this point as to what it is you are seeing.
>>>>>
>>>>> broker - runs on a dedicated box
>>>>>           
>>>>>> qpidd -d --port=5672  --mgmt-enable=yes  --mgmt-pub-interval=1
>>>>>> --auth=no --log-source=yes --log-to-file=/tmp/somename
>>>>>>
>>>>>>             
>>>>> Do you have qpid-tool or anything else running during the test? Does
>>>>> turning management off have any impact on the issue?
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> Apache Qpid - AMQP Messaging Implementation
>>>>> Project:      http://qpid.apache.org
>>>>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>>>>
>>>>>
>>>>>           
>>     
>
>   


Mime
View raw message