qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: Instrumenting the Broker
Date Fri, 11 Jan 2013 10:44:18 GMT
Howdy Lance,

You mention qpid-tool which uses QMF so I'm assuming that you are 
talking about the C++ broker. The Java broker uses JMX and I'm not so 
familiar with how that does instrumentation so the following is only 
really applicable to the C++ broker.


Unfortunately QMF can be a bit non-trivial, it's not helped by the fact 
that there are two variants - QMF1 which is a binary protocol and 
accompanying API and QMF2 which is built on top of Map messages.

The python tools bundled with qpid used to use QMF1, though I *think* 
that as of qpid 0.18 they've started to use QMF2 (trying 0.18 
qpid-config on a 0.8 broker gets a method failed error which suggests 
it's using QMF2 create method - though I've not actually looked at the 
0.18 source).

If you are writing your tools in python there's a whole lot more stuff 
available for QMF.

If you are writing in Java a year back I wrote a complete Java 
implementation of the QMF2 API:
https://cwiki.apache.org/qpid/qmfv2-api-proposal.html

based on the protocol:
https://cwiki.apache.org/qpid/qmf-map-message-protocol.html

The link to my code is here:
https://issues.apache.org/jira/browse/QPID-3675


Slightly frustratingly there hasn't been much feedback on this - I'd 
really like to see it make it into the main code base - I put a huge 
amount of effort into it. The code in that Jira link also contains 
tests/tools that illustrate how to use QMF2 in Java in some non-trivial 
scenarios.


As it happens in a few days time I'm about to release a major uplift to 
that, which will include a QMF2 REST API and most significantly a 
complete HTML5 based web UI that will enable inspection of all QMF 
properties and also queue/exchange/binding adding/deletion so that may 
be just the sort of thing you're interested in - it's pretty close, I'm 
just doing some last minute de-snagging trying to get it looking nice on 
small real-estate mobile browsers.


To more directly answer your question "However, I’m having a hard time 
making that same mapping between the connection stuff and a sender to an 
exchange. " yeah there's some nuance there :-) the main thing is to 
think in terms of UML associations, so for example a binding object has 
queueRef and exchangeRef properties which hold ObjectIds that enable the 
associated queue and exchange to be indexed (binding behaves like a UML 
association class really) what this means though if you want to work out 
how a queue links to an exchange you have to do it from the perspective 
of the binding.

Similarly it's quite involved to find the subscriptions associated with 
a queue because the navigation goes "the wrong way" that is to say 
subscription has a queueRef property.

What I've found is that in general the best (and by far in a way most 
efficient) way to do non-trivial QMF stuff is to to the getObject() 
stuff once for each of the objects that you care about then dereference 
those objects into Maps indexed by objectId - if you do that then you 
can index everything you want easily and cheaply.

If you look at the code for things like qpid-config it's pretty easy to 
get into the trap of doing getObjects() calls on inner loops, which is 
heinously inefficient :-) because each getObjects() call does a 
messaging request/response under the hood.

To get you up and running and give a bit of insight (I hope!!) I've 
attached a couple of python programs I wrote last year (these use QMF1)

connection-audit checks when connections are made and looks up the 
queues being subscribed to against a whitelist, basically if the queue 
being subscribed to is not a queue that's in the whitelist it generates 
a log message.

connection-audit-dom is the same but uses DOM based XML parsing (I 
needed to do that when I had to use it against an oldish python version)

connection-logger does a lot of what I think you care about, basically 
it logs all connections to a broker and provides useful info on them a 
la qpid-config -b queues.


connection-logger-orig was my original version of this which was 
somewhat erm "rushed out" :-) at face value it looks OK for a few queues 
etc. but if you ramp up a load of connections - well try and add a 
couple of hundred queues and you'll see what I meant by heinously 
inefficient above :-D.

connection-logger and connection-logger-orig are functionally equivalent 
so it's worth looking through both to give a bit of insight into the 
right way and wrong way to go about things.

I'm not a python programmer by any stretch of the imagination, but I 
think that they are at least fair examples.

Keep an eye out for my QMF2/UI update over the next few days but 
hopefully this response has given you a decent leg-up.

Best regards,
Frase



On 11/01/13 02:24, Lance D. wrote:
> Hello all.
>
>
> I’m in the process of building QPID inspection tools for a project I’m
> working on.  Now, I’m looking at the results of the qpid-tool.  I see that
> the connection schema has a bunch of useful info (like remote pid, process,
> etc).  I’ve also found ways to link that info up with the queues to figure
> out who’s subscribed to what.  However, I’m having a hard time making that
> same mapping between the connection stuff and a sender to an exchange.
>
>
>
> My question is, is there a way to do that mapping (either with the existing
> tools, or by tapping into the qmf API?
>
>
>
> Thanks,
> -Lance
>


Mime
View raw message