qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: Java QMF library and ObjectId
Date Wed, 17 Sep 2014 20:42:32 GMT
Hi Michal
Epic TL;DR response follows - I'm afraid I got on a roll ..........


You are using QMF to talk to the c++ broker (qpidd) right?

Presumably when you say "the QMF library delivers this update with 
ObjectId of the form..... " you are doing this via the getObjects() 
call? E.g. something like:
         List<QmfConsoleData> queues = 
_console.getObjects("org.apache.qpid.broker", "queue");

Is that correct?

If so the code for that is in
qpid/tools/src/java/qpid-qmf2/src/main/java/org/apache/qpid/qmf2/console/Console.java 
(line 697 on trunk currently).

As far as I can recall that code simply reflects the information in the 
QMF object (e.g. the queue objects in your case) that was sent by the 
broker. It receives the message from JMS via:
                         Message response = 
_responder.receive(timeout*1000);

then subsequently puts the information into a List<QmfConsoleData> via

             if (AMQPMessage.isAMQPList(response))
                         {
                             List<Map> mapResults = 
AMQPMessage.getList(response);
partials.ensureCapacity(partials.size() + mapResults.size());
                             for (Map content : mapResults)
                             {
                                 partials.add(new 
QmfConsoleData(content, agent));
                             }
                         }

That code caters for the fact that JMS doesn't support lists terribly well
qpid/tools/src/java/qpid-qmf2/src/main/java/org/apache/qpid/qmf2/common/AMQPMessage.java 
provides some code to make it a bit easier to deal with Map and List 
messages in java.util Containers.

It basically first decodes the JMS message as a List<Map> because QMF2 
query responses get sent by the Broker as an AMQP List of AMQP Maps. I 
then go through and create a List<QmfConsoleData> where the 
QmfConsoleData is basically a wrapper around a java.util.Map

I've looked at QmfConsoleData and when that gets constructed or 
initialised the ObjectId instance _object_id (which is actually owned by 
the parent QmfDescribed class) gets set via

new ObjectId((Map)m.get("_object_id"));

Which is explicitly getting the property named "_object_id" from the Map 
used to construct the QmfConsoleData, which is as I say a property that 
has been set in the QMF2 Message sent by the broker.


So in a long, roundabout, thinking out loud nutshell I'm pretty sure 
that the "particular reason why the Java implementation of QMF doesn't 
generate more unique ObjectId" is because it's not the Java 
Implementation of QMF that is actually generating the ObjectId in the 
first place, it's simply representing exactly what was sent to it by the 
Broker.



You mention that you've also been using the Python QMF implementation, 
can I ask which one? There are actually two Python QMF implementations 
there's one that lives in:
qpid/extras/qmf/src/py/qmf/console.py

and is used by things like qpid-queue-stats and a few other things. This 
is the asynchronous API and starts something like:
class BrokerManager(Console):

The other python API lives in
qpid/qpid-trunk/qpid/tools/src/py/qpidtoollibs/broker.py

And that's the one used by the python qpid-config etc.

Looking in broker.py getAllQueues calls getAllBrokerObjects which calls 
_doClassQuery which does

     items = []
     done = False
     while not done:
       for item in response.content:
         items.append(item)

Which again appears pretty much to be reflecting what is being sent by 
the broker.

So when you say you've used Python's QMF implementation I'm suspecting 
that you aren't using getAllQueues from qpidtoollibs/broker.py. but are 
in fact using qmf/console.py?


(sorry this is now really TL;DR but I tend to think out loud and the 
play back is I think useful - my Maths lecturer always drummed into me 
"show your working" so it's all his fault :-))

Right....... I've now just proved to my satisfaction that it is indeed 
the behaviour of the Broker. What I've just done is gently hacked the 
qpid-config port that I've just written in JavaScript, that's a good 
"control" because it's independent of either the Java QMF or 
qpidtoollibs and also uses AMQP 1.0 and proton Messenger so should rule 
out any accidental bias.

What I did was hack the queueListRecurse method to look like this:

         for (var i = 0; i < queues.length; i++) {
             var queue = queues[i];
             var queueId = oid(queue._object_id);
console.log("queueId = " + queueId);
.......

So I'm basically printing out the ObjectId that I get directly from the 
QMF response. In my console I see....

<start qpidd here>
./qpid-config.js -r queues
queueId = 
1815:org.apache.qpid.broker:queue:4c44dfde-5ea6-4370-d102-549237ac4ac7_#
Queue '4c44dfde-5ea6-4370-d102-549237ac4ac7_#'
     bind [4c44dfde-5ea6-4370-d102-549237ac4ac7_#] => ''

<that's the oid of the temporary QMF response queue BTW. In another 
window I do qpid-config add queue test>



./qpid-config.js -r queues
queueId = 
1815:org.apache.qpid.broker:queue:b5250fd9-80fc-4208-94e0-8e1bbc41106d_#
Queue 'b5250fd9-80fc-4208-94e0-8e1bbc41106d_#'
     bind [b5250fd9-80fc-4208-94e0-8e1bbc41106d_#] => ''
queueId = 1815:org.apache.qpid.broker:queue:test
Queue 'test'
     bind [test] => ''

<So now you can see the oid of the "test" queue. In the other window I 
then do qpid-config del queue test>


./qpid-config.js -r queues
queueId = 
1815:org.apache.qpid.broker:queue:5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#
Queue '5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#'
     bind [5daeb2a4-11a3-4c4c-a9ff-eee19947a9db_#] => ''

<as you can see, it's gone again. Now I do qpid-config add queue test again>


./qpid-config.js -r queues
queueId = 
1815:org.apache.qpid.broker:queue:f7e8307e-c8ce-441c-8279-f543ad91b076_#
Queue 'f7e8307e-c8ce-441c-8279-f543ad91b076_#'
     bind [f7e8307e-c8ce-441c-8279-f543ad91b076_#] => ''
queueId = 1815:org.apache.qpid.broker:queue:test
Queue 'test'
     bind [test] => ''


Clearly queueId = 1815:org.apache.qpid.broker:queue:test is back and has 
essentially replicated your observation, but it's demonstrably yielding 
what the Broker is telling it and is not a case of the QMF library 
(Java, qpidtoollibs or the JavaScript) itself generating any ObjectID.


What I *think* you are observing in the qmf/console.py library is 
possibly something to do with the old QMF1 messages (IIRC the Broker can 
send both formats) the old QMF1 format was a binary form and the OIDs 
were IIRC quite different in structure (I'm afraid I don't know too much 
about QMF1).



Now that I think about it (sorry it's taken a while until the penny 
finally dropped) the QMF2 specification 
(https://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal) 
actually says something about ObjectIDs and allows them to be "predictable"

"
An object identifier uniquely addresses a data object within the domain 
of its managing agent. QMF represents the object identifier as an opaque 
string. An object identifier can be assigned to the object by the agent 
when the object is instantiated.

Alternatively, a schema can define an object identifier by defining an 
ordered list of names of data items. In this case, the object identifier 
string is built by concatenating the string representations of the value 
of each named data item. This approach is similar to defining index 
fields within a database record.
"

It's the second paragraph that is in fact important for the purposes of 
your observations "concatenating the string representations of the value 
of each named data item" - does this ring any bells: 
1815:org.apache.qpid.broker:queue:test

TBH I should have twigged sooner, I've actually been bitten by this in 
the opposite direction. As it happens I implemented a QMF2 plugin for 
the JavaBroker a while back and I actually originally implemented the 
ObjectIDs by assigning UUIDs, but when I started testing it I noticed it 
didn't work with the Python qpid-config, though it *did* work with my 
Java qpid-config port...

I eventually figured out that my Java and JavaScript qpid-configs make 
no assumptions about the ObjectID, so in order to invoke CRUD methods 
(like in qpid-config add queue test) they query for the "Broker" 
management object and invoke methods on that, but what qpidtoollibs does 
is the following:
"
   def _method(self, method, arguments, 
addr="org.apache.qpid.broker:broker:amqp-broker", timeout=10):
     props = {'method'             : 'request',
              'qmf.opcode'         : '_method_request',
              'x-amqp-0-10.app-id' : 'qmf2'}
     correlator = str(self.next_correlator)
     self.next_correlator += 1

     content = {'_object_id'   : {'_object_name' : addr},
                '_method_name' : method,
                '_arguments'   : arguments}

     message = self.message_class(
       content, reply_to=self.reply_to, correlation_id=correlator,
       properties=props, subject="broker")
     self.tx.send(message)
     response = self.reply_rx.fetch(timeout)
     self.sess.acknowledge()
     if response.properties['qmf.opcode'] == '_exception':
       raise Exception("Exception from Agent: %r" % 
response.content['_values'])
     if response.properties['qmf.opcode'] != '_method_response':
       raise Exception("bad response: %r" % response.properties)
     return response.content['_arguments']
"

In other words it uses "org.apache.qpid.broker:broker:amqp-broker" which 
is actually the ObjectID of the Broker Management Object, so 
qpidtoollibs is choosing not to query for the Broker Management Object 
to retrieve its ObjectID because it knows that the C++ Broker has 
implemented ObjectIDs according to the second paragraph I quoted above.


So in precis:
1) The Java, Python qpidtoollibs and JavaScript QMF2 code are all 
behaving as expected and are not in fact generating their own ObjectIDs 
they are using what the Broker sent them.
2) The Broker is indeed behaving in the way that you observed, but is in 
fact behaving correctly according to the QMF2 Specification.

Sorry it took so long to reach a conclusion that you probably didn't 
want to hear, but hopefully the journey to the conclusion was 
enlightening - and at least shows I'm not making it all up :-D


Sorry this probably doesn't help with your problem. I'm a bit torn - as 
I say I originally implemented OIDs using UUIDs and changed the 
behaviour to be consistent with the C++ Broker and qpidtoollibs, but 
OTOH I suspect that for many people (if not the vast majority - 
excepting you obviously) If you create a queue called "test" it's 
probably actually quite useful to be able to refer to it by name 
org.apache.qpid.broker:queue:test and if you delete it and re-add it 
there's likely a reasonable argument either way as to whether you might 
want it to be the *same* Object (e.g. the same object name in the same 
logical namespace such as a queue and this approach gives your IDs 
temporal persistence even surviving Broker restarts) the other side of 
the coin is that having them unique is also not necessarily an 
unreasonable assumption too - but that I'm afraid is not the way it's 
interpreted by the Broker, and making them unique would itself break 
qpid-config and most likely a bunch of other QMF tools, so it's probably 
not going to gain much traction trying to push for a change in behaviour.



Re "It is quite important for us, since we want to keep track of also 
deleted objects on our side. However, if the QMF library gives us object 
with the same ObjectId (which we already keep as deleted) we have a 
collision. "

I think TBH that you are simply going to have to change your algorithm. 
In practice if you delete a queue named test_queue then re-add the queue 
test_queue you've basically restored exactly the same QMF object as far 
as the Broker is concerned, so technically you don't have a collision 
you have an ObjectID that refers  to the same Object and the deletion of 
test_queue was merely transient behaviour.

Now if you actually *care* that the queue has been temporarily deleted 
(perhaps for auditing purposes or whatever) then you have a few options 
- but as far as the Broker is concerned they are the *same* Object.

If you want to identify that this has occurred you need to bear in mind 
that although you say "we are using Java QMF library for receiving 
asynchronously broker's objects updates " if you are using the 
getObjects() stuff you are not actually working asynchronously you are 
infact polling.

To deal more asynchronously you could either use QMF2 Events (which are 
delivered asynchronously) and either use the info from the Event or use 
that to trigger a call to getObjects() - look in
qpid/qpid-trunk/qpid/tools/src/java/qpid-qmf2-tools/src/main/java/org/apache/qpid/qmf2/tools

The ConnectionAudit.java and ConnectionLogger.java examples both 
basically receive QMF Events
         if (wi instanceof EventReceivedWorkItem)
.....

Then subsequently go on to do a bunch of getObjects() calls, so you 
would see IIRC queueDeclare/queueDelete events that would relate to your 
"test_queue"


Alternatively you could make use of the QMF2 API "Query Subscription" 
mechanism. If you look in QpidQueueStats.java I've given an example of 
how to use that, now I *personally* wouldn't go down that route. 
Although my Java QMF2 API has implemented everything specified in 
https://cwiki.apache.org/confluence/display/qpid/QMFv2+API+Proposal (as 
you may have gathered I'm just a bit OCD) as it happens the Java QMF 
implementation is actually the only API that ever actually got round to 
implementing all of that...... what basically happened was the 
realisation that the QMF2 protocol is really when all is said and done 
just a bunch of Map Messages, so there's a bit of an argument whether 
one should use an API or simply just use the Map Messages. My view is 
that there's more of an argument for an API in strongly typed languages 
like Java and C++ (most of QmfConsoleData is actually defensive code 
that I discovered the hard way 'cause qpidd sends strings as AMQP byte 
and I got killed by ClassCastException when I tried to cast byte[] to 
String :o)). It's also woth pointing out that there's likely to be a (I 
suspect vaguely slow) migration from QMF to AMQP 1.0 Management 
Specification and at some point when I get my act together I'll probably 
try to provide a mechanism to get info from both protocols to ease the 
transition, at which point I *might* deprecate some of the more 
fancy/fiddly features of the QMF2 API 'cause I suspect that nobody has 
actually used them since I wrote that demo...... If you stick to 
getObjects and the Event stuff I'm more likely to try to make that 
polyglot than the Query Subscription stuff, which was flipping fiddly to 
implement if I'm honest.


I really hope that this has been useful even if not quite what you'd 
hoped for, sorry it ended up as "War and Peace", but as a result you now 
probably know about as much as anyone does on QMF :->

Finally, although I don't mind answering PM'd questions I'd be grateful 
if you could post any other qpid related questions to the user list - 
it's mainly because there's a fighting change that if you have a Qpid 
related question you won't be the only one who might like to know the 
answer.

Cheers,
Frase


On 17/09/14 12:07, michal.zerola@gmail.com wrote:
> Hello Fraser,
>
> we are using Java QMF library for receiving asynchronously broker's objects updates.
I have a question regarding the ObjectId - which is and object identifying the broker's object
change.
>
> Imagine I create a queue test_queue on the broker, the QMF library delivers this update
with ObjectId of the form (toString()):
>
> @2@org.apache.qpid.broker:queue:test_queue
>
> Now I delete the queue and I get the update identified again with the same ObjectId.
So far so good. But if I create the queue again (with the same name), QMF library sends me
the update again with the same ObjectId. This is different comparing to e.g. Python's QMF
implementation, where I would get the new ID.
>
> It is quite important for us, since we want to keep track of also deleted objects on
our side. However, if the QMF library gives us object with the same ObjectId (which we already
keep as deleted) we have a collision.
>
> So questions from my side:
> - Is there a particular reason why the Java implementation of QMF doesn't generate more
unique ObjectId (like Python implementation does)?
> - Would it be possible to change this behavior? We can implement it by ourselves and
provide the patch to the Qpid community if you can give us brief instructions on it.
>
> Thank you in advance!
>
> Best regards,
>
> Michal
>
>
> _____________________________________
> Sent from http://qpid.2158936.n2.nabble.com
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message