qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Trieloff <cctriel...@redhat.com>
Subject Re: Subscriptions slows down broker (was Memory pile up on broker)
Date Mon, 16 Mar 2009 00:39:16 GMT

I don't believe any profiling has been done on the headers exchange. If 
you are on Linux, run oprofile on it
and either create a patch to correct the hot spot or mail the profile to 
the list.

if needed I can also give mail you a stap (system tap) script for lock 
analysis for the headers exchange.

GS.Chandra N wrote:
> Gordon, Tedd
> After the patches to the headersexchange.cpp was applied, i can now 
> clearly see the incoming rates with the help of the tool Tedd sent.
> The rates are at about 90-100 odd messages per sec (a drop from 5200 
> odd without subscriptions).
> Sort of matches my earlier observations using qpid-tool. What next?
> Do you think there might be any way to add the headers exchange 
> performance improvements to your list of priority fixes? I'm not on 
> the dev forums so i 'm not sure if the issue is communicated out there.
> I will try to test the multiple subscriptions XML exchange next to if 
> that improves matters in any respect.
> Thanks for the great support in fully un-reavelling the issue.
> Rgds
> gs
> ps : I can also still reproduce the memory pile up by blocking the 
> broker using qpid-tool and setting the mgmgt-pub interval to 1 which 
> of-course i can avoid by not letting qpid-tool stick around and 
> increasing the mgmt-pub interval to 10 or more.
> On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <gs.chandran.n@gmail.com 
> <mailto:gs.chandran.n@gmail.com>> wrote:
>     Hi,
>     I found that if i kept the qpid-tool open long enough all my
>     clients and publishers would throw exceptions and come out.
>     Earlier when i was performing my tests i had 3 putty windows
>     opened to all the 3 servers and it seems as if, i was keeping
>     qpid-tool opened just long enough to cause build up but closing it
>     to run top and so on a so forth such that the memory piled up but
>     did not cause timeouts at the client either.
>     Observations
>     1.When there are subscriptions and a qpid-tool around messages
>     seem to pile up. Why is this so? *
>     *
>     2. When there are no subscribers and no qpid-tool the publishing
>     processes cause the CPu to go to 90% at the publishing box. But
>     when i started up subscribers, the publishing box CPU fell to 0%
>     and all the python processes piled up memory.
>     Concern : I'm not able to find out here if the publishers are
>     still able to send out messages at the speed that it should when
>     subscriptions are around. Or perhaps the broker is not able to
>     pull enough messages - iam not sure which of this is happeneing.
>     Probably latter? How do i tell?
>     If i use qpid-tool to connect to the broker at this point, every
>     time i hit show exchange i get the same stats with the same time
>     stamp. it looks like the broker is too busy trying to match
>     subscriptions even to refresh the stats.   
>     3.Concern : Once the subscriptions are made, the CPU at the broker
>     box* *(high end dual core XEON with ht) goes to 90%.  Even at this
>     level, i'm not sure the broker is able to match and discard all
>     the messages fast enough (due to 2nd observation). Or how do i tell? 
>     Thanks
>     gs
>     ps : Is there a sure shot way to find out the message rates ? (I
>     currently use qpid-tool show exchange command to find the no of
>     msgRecieves and divide by time-dfference from 2 times the command
>     is run)
>     On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N
>     <gs.chandran.n@gmail.com <mailto:gs.chandran.n@gmail.com>> wrote:
>         I'm able to reproduce the issue, when i keep qpid-tool open
>         all the time with the mgmt switches.
>         However i'm not able to do so if qpid-tool is not open (did
>         not try with qpid-tool and no mgmt switches).
>         Thanks
>         gs
>         On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <gsim@redhat.com
>         <mailto:gsim@redhat.com>> wrote:
>             GS.Chandra N wrote:
>                 ps : Please find attached the scripts that create the
>                 issue
>             I'm afraid I wasn't able to recreate the issue with your
>             tests. Memory stayed reasonable and constant as the test
>             was running (5 servers, 5 clients).
>             I'm baffled at this point as to what it is you are seeing.
>                 broker - runs on a dedicated box
>                 qpidd -d --port=5672  --mgmt-enable=yes
>                  --mgmt-pub-interval=1 --auth=no --log-source=yes
>                 --log-to-file=/tmp/somename
>             Do you have qpid-tool or anything else running during the
>             test? Does turning management off have any impact on the
>             issue?
>             ---------------------------------------------------------------------
>             Apache Qpid - AMQP Messaging Implementation
>             Project:      http://qpid.apache.org
>             Use/Interact: mailto:users-subscribe@qpid.apache.org
>             <mailto:users-subscribe@qpid.apache.org>

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message