qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: Instrumenting the Broker
Date Fri, 11 Jan 2013 15:15:58 GMT
On 11/01/13 13:11, Lance D. wrote:
> Wow.  Thanks for the huge amount of detail!
No worries, hope it helps. One of the best things about Qpid is the 
strong user community, I've got a lot out of it so it's nice to be able 
to give a little back.
>
> You are correct: I am using the C++ broker.  As far as language for my
> tool, I'm pretty agnostic; I want a tool that suits my needs and I've got
> the resources to do it in any viable language (though I am partial to C++).
Unfortunately C++ has the poorest QMF support IMHO :-( There is an API, 
but it bears no resemblance to the QMF2 API that I linked previously :-( 
I've not done much with that I'm afraid. I keep promising myself that 
I'll port my Java implementation to C++, but until I get more feedback 
on my Java stuff it's hard to find the motivation.....

The is an example using the C++ QMF API in 
qpid/cpp/bindings/qmf2/examples/cpp but as I say I'm not overly familiar 
with that API and haven't used it myself.

I've attached a few examples that Gordon Sim put together that use the 
QMF2 protocol directly from C++ which you might find useful.

Java and python currently have the best QMF support, though as I said 
previously I'll shortly be releasing a REST API for QMF2 and also 
JavaScript support :-)

>
> I'm going to follow up on one paragraph in particular:
> "To more directly answer your question "However, I’m having a hard time
> making that same mapping between the connection stuff and a sender to an
> exchange. " yeah there's some nuance there :-) the main thing is to think
> in terms of UML associations, so for example a binding object has queueRef
> and exchangeRef properties which hold ObjectIds that enable the associated
> queue and exchange to be indexed (binding behaves like a UML association
> class really) what this means though if you want to work out how a queue
> links to an exchange you have to do it from the perspective of the binding."
>
> My problem is that my publishers establish a 'send only' connection, so
> they link to the exchange directly (without a queue).  To put it another
> way, if I have n publishers sending to exchange 'myExchange', when I run
> 'qpid-stat -q' I don't see n queues bound to myExchange.  So, without a
> queueRef, I have no binding object for those publishers, and no way to get,
> for example, the remote PID.  Am I missing something?
Ahhh so what I think you're saying is that you want to figure out 
linkage from producer connections.

Unfortunately I don't think that you're missing something. So as I said 
previously it's possible (though not trivial) to work out associations 
between consumer connections and the queues that they are getting their 
data from, but there isn't any good way to do what might reasonably be 
considered the converse which is to work out the exchange that a 
producer connection is publishing to.

I asked about that *ages* ago - I might be able to find the post in my 
sent box but it'll probably take a while. IIRC the response that I got 
from Gordon Sim was that it related to the AMQP spec and that there 
wasn't a mechanism to do this.

*So basically I don't think that there's any way to look up an exchange 
associated with a connection.*

The closest that I've got to that is to at least flag up that a 
connection relates to a "producer only" client by checking for 
connections that have sessions but no subscribers.

The remotePid is a property of the connection management object.

if you only have producer clients (isn't that a bit useless with nothing 
consuming the messages??) all you will see in that part of the graph is 
connections and sessions.


I wish that I could give a happier answer, but I'm close to certain that 
you aren't missing something. I've had users ask for this sort of thing 
myself and have had to say it's not possible, I've not even worked out a 
reliable way to even *infer* the exchange. As I say the closest I got 
was to at least work out which of my connections only related to 
producers so I could work out the remote host and pid of those.

Perhaps Gordon or one of the others on the core Qpid team has some more 
insight, but I suspect I've probably done more with QMF than most.

Sorry I can't give the answer you'd like,
Frase

>
> Thanks again,
> -Lance
>
> On Fri, Jan 11, 2013 at 5:44 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
>> wrote:
>> Howdy Lance,
>>
>> You mention qpid-tool which uses QMF so I'm assuming that you are talking
>> about the C++ broker. The Java broker uses JMX and I'm not so familiar with
>> how that does instrumentation so the following is only really applicable to
>> the C++ broker.
>>
>>
>> Unfortunately QMF can be a bit non-trivial, it's not helped by the fact
>> that there are two variants - QMF1 which is a binary protocol and
>> accompanying API and QMF2 which is built on top of Map messages.
>>
>> The python tools bundled with qpid used to use QMF1, though I *think* that
>> as of qpid 0.18 they've started to use QMF2 (trying 0.18 qpid-config on a
>> 0.8 broker gets a method failed error which suggests it's using QMF2 create
>> method - though I've not actually looked at the 0.18 source).
>>
>> If you are writing your tools in python there's a whole lot more stuff
>> available for QMF.
>>
>> If you are writing in Java a year back I wrote a complete Java
>> implementation of the QMF2 API:
>> https://cwiki.apache.org/qpid/**qmfv2-api-proposal.html<https://cwiki.apache.org/qpid/qmfv2-api-proposal.html>
>>
>> based on the protocol:
>> https://cwiki.apache.org/qpid/**qmf-map-message-protocol.html<https://cwiki.apache.org/qpid/qmf-map-message-protocol.html>
>>
>> The link to my code is here:
>> https://issues.apache.org/**jira/browse/QPID-3675<https://issues.apache.org/jira/browse/QPID-3675>
>>
>>
>> Slightly frustratingly there hasn't been much feedback on this - I'd
>> really like to see it make it into the main code base - I put a huge amount
>> of effort into it. The code in that Jira link also contains tests/tools
>> that illustrate how to use QMF2 in Java in some non-trivial scenarios.
>>
>>
>> As it happens in a few days time I'm about to release a major uplift to
>> that, which will include a QMF2 REST API and most significantly a complete
>> HTML5 based web UI that will enable inspection of all QMF properties and
>> also queue/exchange/binding adding/deletion so that may be just the sort of
>> thing you're interested in - it's pretty close, I'm just doing some last
>> minute de-snagging trying to get it looking nice on small real-estate
>> mobile browsers.
>>
>>
>> To more directly answer your question "However, I’m having a hard time
>> making that same mapping between the connection stuff and a sender to an
>> exchange. " yeah there's some nuance there :-) the main thing is to think
>> in terms of UML associations, so for example a binding object has queueRef
>> and exchangeRef properties which hold ObjectIds that enable the associated
>> queue and exchange to be indexed (binding behaves like a UML association
>> class really) what this means though if you want to work out how a queue
>> links to an exchange you have to do it from the perspective of the binding.
>>
>> Similarly it's quite involved to find the subscriptions associated with a
>> queue because the navigation goes "the wrong way" that is to say
>> subscription has a queueRef property.
>>
>> What I've found is that in general the best (and by far in a way most
>> efficient) way to do non-trivial QMF stuff is to to the getObject() stuff
>> once for each of the objects that you care about then dereference those
>> objects into Maps indexed by objectId - if you do that then you can index
>> everything you want easily and cheaply.
>>
>> If you look at the code for things like qpid-config it's pretty easy to
>> get into the trap of doing getObjects() calls on inner loops, which is
>> heinously inefficient :-) because each getObjects() call does a messaging
>> request/response under the hood.
>>
>> To get you up and running and give a bit of insight (I hope!!) I've
>> attached a couple of python programs I wrote last year (these use QMF1)
>>
>> connection-audit checks when connections are made and looks up the queues
>> being subscribed to against a whitelist, basically if the queue being
>> subscribed to is not a queue that's in the whitelist it generates a log
>> message.
>>
>> connection-audit-dom is the same but uses DOM based XML parsing (I needed
>> to do that when I had to use it against an oldish python version)
>>
>> connection-logger does a lot of what I think you care about, basically it
>> logs all connections to a broker and provides useful info on them a la
>> qpid-config -b queues.
>>
>>
>> connection-logger-orig was my original version of this which was somewhat
>> erm "rushed out" :-) at face value it looks OK for a few queues etc. but if
>> you ramp up a load of connections - well try and add a couple of hundred
>> queues and you'll see what I meant by heinously inefficient above :-D.
>>
>> connection-logger and connection-logger-orig are functionally equivalent
>> so it's worth looking through both to give a bit of insight into the right
>> way and wrong way to go about things.
>>
>> I'm not a python programmer by any stretch of the imagination, but I think
>> that they are at least fair examples.
>>
>> Keep an eye out for my QMF2/UI update over the next few days but hopefully
>> this response has given you a decent leg-up.
>>
>> Best regards,
>> Frase
>>
>>
>>
>>
>> On 11/01/13 02:24, Lance D. wrote:
>>
>>> Hello all.
>>>
>>>
>>> I’m in the process of building QPID inspection tools for a project I’m
>>> working on.  Now, I’m looking at the results of the qpid-tool.  I see that
>>> the connection schema has a bunch of useful info (like remote pid,
>>> process,
>>> etc).  I’ve also found ways to link that info up with the queues to figure
>>> out who’s subscribed to what.  However, I’m having a hard time making that
>>> same mapping between the connection stuff and a sender to an exchange.
>>>
>>>
>>>
>>> My question is, is there a way to do that mapping (either with the
>>> existing
>>> tools, or by tapping into the qmf API?
>>>
>>>
>>>
>>> Thanks,
>>> -Lance
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>


Mime
View raw message