qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lance D." <lan...@gmail.com>
Subject Re: Instrumenting the Broker
Date Fri, 11 Jan 2013 13:11:21 GMT
Wow.  Thanks for the huge amount of detail!

You are correct: I am using the C++ broker.  As far as language for my
tool, I'm pretty agnostic; I want a tool that suits my needs and I've got
the resources to do it in any viable language (though I am partial to C++).

I'm going to follow up on one paragraph in particular:
"To more directly answer your question "However, I’m having a hard time
making that same mapping between the connection stuff and a sender to an
exchange. " yeah there's some nuance there :-) the main thing is to think
in terms of UML associations, so for example a binding object has queueRef
and exchangeRef properties which hold ObjectIds that enable the associated
queue and exchange to be indexed (binding behaves like a UML association
class really) what this means though if you want to work out how a queue
links to an exchange you have to do it from the perspective of the binding."

My problem is that my publishers establish a 'send only' connection, so
they link to the exchange directly (without a queue).  To put it another
way, if I have n publishers sending to exchange 'myExchange', when I run
'qpid-stat -q' I don't see n queues bound to myExchange.  So, without a
queueRef, I have no binding object for those publishers, and no way to get,
for example, the remote PID.  Am I missing something?

Thanks again,
-Lance

On Fri, Jan 11, 2013 at 5:44 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:

> Howdy Lance,
>
> You mention qpid-tool which uses QMF so I'm assuming that you are talking
> about the C++ broker. The Java broker uses JMX and I'm not so familiar with
> how that does instrumentation so the following is only really applicable to
> the C++ broker.
>
>
> Unfortunately QMF can be a bit non-trivial, it's not helped by the fact
> that there are two variants - QMF1 which is a binary protocol and
> accompanying API and QMF2 which is built on top of Map messages.
>
> The python tools bundled with qpid used to use QMF1, though I *think* that
> as of qpid 0.18 they've started to use QMF2 (trying 0.18 qpid-config on a
> 0.8 broker gets a method failed error which suggests it's using QMF2 create
> method - though I've not actually looked at the 0.18 source).
>
> If you are writing your tools in python there's a whole lot more stuff
> available for QMF.
>
> If you are writing in Java a year back I wrote a complete Java
> implementation of the QMF2 API:
> https://cwiki.apache.org/qpid/**qmfv2-api-proposal.html<https://cwiki.apache.org/qpid/qmfv2-api-proposal.html>
>
> based on the protocol:
> https://cwiki.apache.org/qpid/**qmf-map-message-protocol.html<https://cwiki.apache.org/qpid/qmf-map-message-protocol.html>
>
> The link to my code is here:
> https://issues.apache.org/**jira/browse/QPID-3675<https://issues.apache.org/jira/browse/QPID-3675>
>
>
> Slightly frustratingly there hasn't been much feedback on this - I'd
> really like to see it make it into the main code base - I put a huge amount
> of effort into it. The code in that Jira link also contains tests/tools
> that illustrate how to use QMF2 in Java in some non-trivial scenarios.
>
>
> As it happens in a few days time I'm about to release a major uplift to
> that, which will include a QMF2 REST API and most significantly a complete
> HTML5 based web UI that will enable inspection of all QMF properties and
> also queue/exchange/binding adding/deletion so that may be just the sort of
> thing you're interested in - it's pretty close, I'm just doing some last
> minute de-snagging trying to get it looking nice on small real-estate
> mobile browsers.
>
>
> To more directly answer your question "However, I’m having a hard time
> making that same mapping between the connection stuff and a sender to an
> exchange. " yeah there's some nuance there :-) the main thing is to think
> in terms of UML associations, so for example a binding object has queueRef
> and exchangeRef properties which hold ObjectIds that enable the associated
> queue and exchange to be indexed (binding behaves like a UML association
> class really) what this means though if you want to work out how a queue
> links to an exchange you have to do it from the perspective of the binding.
>
> Similarly it's quite involved to find the subscriptions associated with a
> queue because the navigation goes "the wrong way" that is to say
> subscription has a queueRef property.
>
> What I've found is that in general the best (and by far in a way most
> efficient) way to do non-trivial QMF stuff is to to the getObject() stuff
> once for each of the objects that you care about then dereference those
> objects into Maps indexed by objectId - if you do that then you can index
> everything you want easily and cheaply.
>
> If you look at the code for things like qpid-config it's pretty easy to
> get into the trap of doing getObjects() calls on inner loops, which is
> heinously inefficient :-) because each getObjects() call does a messaging
> request/response under the hood.
>
> To get you up and running and give a bit of insight (I hope!!) I've
> attached a couple of python programs I wrote last year (these use QMF1)
>
> connection-audit checks when connections are made and looks up the queues
> being subscribed to against a whitelist, basically if the queue being
> subscribed to is not a queue that's in the whitelist it generates a log
> message.
>
> connection-audit-dom is the same but uses DOM based XML parsing (I needed
> to do that when I had to use it against an oldish python version)
>
> connection-logger does a lot of what I think you care about, basically it
> logs all connections to a broker and provides useful info on them a la
> qpid-config -b queues.
>
>
> connection-logger-orig was my original version of this which was somewhat
> erm "rushed out" :-) at face value it looks OK for a few queues etc. but if
> you ramp up a load of connections - well try and add a couple of hundred
> queues and you'll see what I meant by heinously inefficient above :-D.
>
> connection-logger and connection-logger-orig are functionally equivalent
> so it's worth looking through both to give a bit of insight into the right
> way and wrong way to go about things.
>
> I'm not a python programmer by any stretch of the imagination, but I think
> that they are at least fair examples.
>
> Keep an eye out for my QMF2/UI update over the next few days but hopefully
> this response has given you a decent leg-up.
>
> Best regards,
> Frase
>
>
>
>
> On 11/01/13 02:24, Lance D. wrote:
>
>> Hello all.
>>
>>
>> I’m in the process of building QPID inspection tools for a project I’m
>> working on.  Now, I’m looking at the results of the qpid-tool.  I see that
>> the connection schema has a bunch of useful info (like remote pid,
>> process,
>> etc).  I’ve also found ways to link that info up with the queues to figure
>> out who’s subscribed to what.  However, I’m having a hard time making that
>> same mapping between the connection stuff and a sender to an exchange.
>>
>>
>>
>> My question is, is there a way to do that mapping (either with the
>> existing
>> tools, or by tapping into the qmf API?
>>
>>
>>
>> Thanks,
>> -Lance
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message