qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fraser Adams <fraser.ad...@blueyonder.co.uk>
Subject Re: New Feature Announce - Java QMF2 API Implementation.
Date Thu, 05 Jan 2012 12:28:55 GMT
Hi Ted,
That's great - I must admit I was getting a bit paranoid as I've not had 
any feedback on this.

I think that it is quite a way ahead of the C++ QMF2 stuff, unless 
things have changed recently the C++ QMF2 stuff didn't seem to follow 
the QMF2 API at all, though clearly it was using the QMF2 protocol.

I'd tend to agree with yourself and the others that it's a good idea to 
separate QMF from the Qpid namespace and consider QMF to be a separate 
and layered capability that adds value to Qpid. That's exactly the case 
with the Java API implementation, which is really to all intents and 
purposes just a JMS application.

My one reservation is that if QMF got separated there may be a danger 
that it could become a bit of a second class citizen, I'd be very keen 
to avoid that happening. As it happens one of the reasons I started down 
the path of the Java QMF2 API implementation was because the original 
Java QMF stuff seemed very broken (and it was mentioned that it was not 
supported/maintained and up for removal from the code base). Given my 
experience with the QMF2 stuff I suspect where the original Java QMF 
stuff is most likely to be broken is in the String handling. I've 
mentioned in several threads lots of nasty interoperability issues due 
to C++ binary strings (as opposed to utf8 encoded strings), which get 
mapped into byte[]. I've put in a lot of defensive coding to cater for this.

Another thing that my implementation added to the QMF2 API was 
overloaded methods on addConnection() so I can add additional 
information to the address that gets created e.g.

  _console.addConnection(connection, " ; {link: 
{name:'big-payload-console', x-declare: {arguments: {'qpid.policy_type': 
ring, 'qpid.max_size': 500000000}}}}");

This adds to the generality of QMF2, the idea here is that the QMF2 
Agent and Console simply become a message based service invocation 
framework, with all of the Agent/Method etc. discovery built in. Using 
the extended addConnection() I can send large payloads. I can also set 
explicit names so, for example if I had a service that may take a very 
long time to process I can actually set it up so responses are delivered 
down a named persistent queue so in theory my console could go away and 
come back later to retrieve the asynchronous response.

On a (related) aside I've also written a patch for the broker to support 
QMF2 query requests for types other than OBJECT see:

https://issues.apache.org/jira/browse/QPID-3696


When I was working on this patch it uncovered a couple of bugs in the 
QMF2 API that I'm in the process of fixing. In actual fact they relate 
to a bit of defensive coding that needed to be added where if a schema 
Map got returned without a _properties or _methods key I got a null 
pointer exception. In theory this shouldn't happen as they are mandatory 
properties but I got bitten as I was putting together the broker patch 
so it's right to add some checks. My actual finished broker patch 
doesn't cause this problem as I've made sure that I included all of the 
mandatory properties.

One other issue that I've uncovered is that if I do a getObjects() where 
there are lots of objects (I think > 100) then the broker actually sends 
multiple messages with everyone but the last makred with a "partial" 
header. I only noticed this when I was writing the query patch so I need 
to add support for this to the QMF2 API. It's actually what I'm working 
on at the moment so I'll update the Jira when I've added this stuff in.

It would be good if you could have a play with this stuff, I've been 
working on it most weekends since June so there's quite a lot of it. I 
think that I've covered most of the obvious gotchas, but clearly with 
this much code there are bound to be bugs.

Cheers,
Frase



On 04/01/12 13:53, Ted Ross wrote:
> Frase,
>
> Sorry for my delayed response to this.  I've fallen a little behind on 
> the users list.
>
> Thank you for this contribution.  From your description, it looks as 
> though it's a little ahead of the C++ counterpart!  A Java 
> implementation of QMF2 is very desirable and valuable so it should 
> find its way into the code base so it can become part of the released 
> software.
>
> I'm going to send an email to dev and users later today to propose 
> that we separate QMF from the Qpid namespace (i.e. consider QMF to be 
> a separate and layered capability that adds value to Qpid) and it 
> looks like your Java implementation should be included as well.
>
> -Ted
>
> On 12/11/2011 10:52 AM, Fraser Adams wrote:
>> Hello All,
>> By way of giving something back to the Qpid community I've (finally) 
>> got my implementation of the QMF2 API (which for those not already 
>> familiar is described here 
>> https://cwiki.apache.org/qpid/qmfv2-api-proposal.html) into a form 
>> that is hopefully generally usable.
>>
>> It's taken a bit longer than I'd have liked, but I've spent the time 
>> adding some example tools that are based on the API. Hopefully these 
>> tools are useful, at the very least they provide a fair illustration 
>> of the QMF2 API including such things as method invocation and Query 
>> Subscription.
>>
>> There's quite a lot of if (just short of 100 classes including tests 
>> and tools) so I've zipped it up and attached it to a Jira.
>>
>> https://issues.apache.org/jira/browse/QPID-3675
>>
>> Note that there are some quirks, especially if you use a version of 
>> Qpid < 0.12. Older versions require a patch to allow the app-id 
>> property name to be set. The patch is included in the distro and the 
>> shell scripts that launch the tests and tools add the patch jar to 
>> the front of the class path. I've tested with Qpid 0.8, 0.10 and 0.12.
>>
>> Note too that some of the features won't work with Qpid 0.8. In 
>> particular the QMF2 create and delete methods weren't added until 
>> Qpid 0.10 so the add queue/add binding etc. features of QpidConfig 
>> won't work with Qpid 0.8, similarly for QpidCtrl.
>>
>> The code contains a fair amount of comment and JavaDoc so hopefully 
>> is should be fairly easy to use.
>>
>> I certainly can't guarantee it'll be bug free :-D but I've tried 
>> reasonably hard to break things, so at the very least it should be 
>> robust enough to be usable. ConnectionAudit and QpidQueueStats seem 
>> to reconnect and survive broker restarts pretty well.
>>
>> This is the first release of a QMF2 API Implementation for Java.
>>
>> Features:
>>
>>    * Full Implementation of QMF2 Console, Agent and AgentExternal.
>>    * Supports QMF2 Query Subscriptions on Agent/AgentExternal
>>      implementations.
>>    * Emulates QMF2 Query Subscriptions on the Console side for the
>>      broker ManagementAgent by
>>      intercepting _data indications and filtering against QmfQuery.
>>      This is necessary as the
>>      ManagemetAgent doesn't yet support QMF2 style Query Subscriptions.
>>    * Console supports Agent discovery via findAgent() method, which can
>>      support partial matches,
>>      which is useful because the full Agent name has a UUID
>>      representing the "instance" so it's hard
>>      to know the full name.
>>    * QmfQuery supports regex matching.
>>    * Supports QMF2 WorkItem Event model and in addition supports an
>>      alternative QmfEventListener
>>      Event model, which is rather more like the JMS MessageListener 
>> model.
>>
>> Example Tools Provided:
>>
>>    * ConnectionAudit: Audits connections to one or more Qpid message
>>      brokers against a whitelist.
>>    * ConnectionLogger: A QMF2 class used to provide information about
>>      connections made to a broker.
>>    * QpidConfig: QpidConfig is a fairly "literal" Java port of the
>>      python qpid-config tool. Uses pure
>>      QMF2 for adding/deleting queues, exchanges & bindings this
>>      provides useful illustration of how
>>      to do these things using the ManagementAgent method calls.
>>    * QpidCtrl: A tool to allow QMF2 methods to be invoked from the
>>      command line.
>>    * QpidPrintEvents: Collect and print events from one or more Qpid
>>      message brokers.
>>    * QpidQueueStats: Collect and print queue statistics. This is a
>>      rewrite of the Python version and
>>      illustrates the use of QuerySubscriptions (via the Console side
>>      emulation)to subscribe to
>>      objects on the ManagementAgent.
>>    * QueueFuse: QueueFuse provides protection to message producers from
>>      consumers who can't consume
>>      messages fast enough.
>>
>> A couple of utility classes are provided too:
>>
>>    * GetOpt: Is an implementation of the Python GetOpt, which makes
>>      command line parsing a bit less of a pain.
>>    * ConnectionHelper: Provides support for a variety of different URL
>>      formats. As it happens the C++ AMQP URL, the Java ConnectionURL
>>      and the Python tool "BrokerURL" are all slightly different.
>>      ConnectionHelper tries to support them all as well as a prototype
>>      for the proposed new AMQP URL
>>      https://cwiki.apache.org/qpid/url-format-proposal.html. It's
>>      obviously still possible to create JMS Connections for QMF2 via
>>      JNDI, but ConnectionHelper makes it a bit easier for command line
>>      based tools where we may want to connect to lots of different
>>      broker instances.
>>
>>
>> I hope that this is useful, I certainly find it useful as I need to 
>> do a lot with broker management info.
>>
>> It'll be really nice if this makes it into an official Qpid release 
>> at some point.
>>
>> Enjoy!!
>> Best Regards,
>> Frase
>>
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project:      http://qpid.apache.org
>> Use/Interact: mailto:users-subscribe@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org
>


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org


Mime
View raw message