Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
Received-SPF: pass (athena.apache.org: domain of fraser.adams@blueyonder.co.uk
 designates 81.103.221.49 as permitted sender)
Message-ID: <515C83B8.4050000@blueyonder.co.uk>
Date: Wed, 03 Apr 2013 20:32:08 +0100
From: Fraser Adams <fraser.adams@blueyonder.co.uk>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: users@qpid.apache.org
Subject: Re: Questions from a novice
References: 
 <CAB+Aj0t4z6mRXZii=bV8bwBoS5FAfbsgmJR6GZJM8zkTFDHaVg@mail.gmail.com>
 <5151DD39.4040400@redhat.com>
 <CAB+Aj0vHSNY8pjoeOtcxwQT8ackunYDUZTDhMVDjdi-e6_BksA@mail.gmail.com>
 <51559722.7000604@redhat.com>
In-Reply-To: <51559722.7000604@redhat.com>
Content-Type: multipart/alternative;
 boundary="------------040605000801050108000601"

--------------040605000801050108000601
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi folks,
I'm revisiting this having got my IT back, comments inline...

Frase

On 29/03/13 13:29, Gordon Sim wrote:
> On 03/28/2013 08:52 PM, Bill Freeman wrote:
>> So it occurs to me to ask whether the correspondence between named queue
>> and object id survives broker restarts?
>
> I believe with QMFv2 and qpidd, the object identifiers for queues are 
> formed from the package, the class and the queue's name. So although 
> the management id is not persisted, it would be the same on recovery 
> (as it is essentially the queue name).
According to the protocol spec 
https://cwiki.apache.org/qpid/qmf-map-message-protocol.html ObjectId is 
formed thus:

   OBJECT_ID := { _agent_name:  STRING,
                  _agent_epoch: NUMBER,
                  _object_name: STRING
                }

_agent_name 	yes 	Name of the agent that is managing the referenced data
_agent_epoch 	yes 	Numeric epoch of the agent process. This number is 
managed by the agent and is incremented each time the agent process 
starts. This field is only present for /transient/ object IDs that must 
not be the same for a given object across an agent restart. /Persistent/ 
object IDs must not include this field.
_object_name 	no 	Name of the data that uniquely identifies the data 
within the context of the agent.


According to the QMF2 API spec 
https://cwiki.apache.org/qpid/qmfv2-api-proposal.html "An object 
identifier uniquely addresses a data object within the domain of its 
managing agent. QMF represents the object identifier as an opaque string"

Clearly there's a bit of ambiguity between "opaque string" and the 
fields specified in the protocol :-)

It's even weirder because despite the alleged opacity there's ambiguity 
because the API has in SchemaObjectClass :

       .set_id_names([name-list]): set the value of the order list of names to use when constructing the object identifier.


In other words that's a mechanism used to set the _object_name. In my 
Java Implementation for Agent I've got code that does:

         ObjectId addr = object.getObjectId();
         if (addr == null)
         {
             SchemaClassId classId = object.getSchemaClassId();
             SchemaClass schema = _schemaCache.get(classId);

             // Try to create an objectName using the property names 
that have been specified as idNames in the schema
             StringBuilder buf = new StringBuilder();
             // Initialise idNames as an empty array as we want to check 
if a key has been used to construct the name.
             String[] idNames = {};
             if (schema != null && schema instanceof SchemaObjectClass)
             {
                 idNames = ((SchemaObjectClass)schema).getIdNames();
                 for (String property : idNames)
                 {
                     buf.append(object.getStringValue(property));
                 }
             }
             String objectName = buf.toString();

             // If the schema hasn't given any help we use a UUID. Note 
that we check the length of idNames too
             // as a given named key property might legitimately be an 
empty string (e.g. the default direct
             // exchange has name == "")
             if (objectName.length() == 0 && idNames.length == 0) 
objectName = UUID.randomUUID().toString();

             // Finish up the name by incorporating package and class names
             objectName = classId.getPackageName() + ":" + 
classId.getClassName() + ":" + objectName;

             // Now we've got a good name for the object we create its 
ObjectId and add that to the object
             addr = new ObjectId(_name, objectName, _epoch);

             object.setObjectId(addr);
         }

So if I do setObjectId() on a QmfConsoleData instance before I call 
addObject() it will explicitly use the ID I've set (I needed to do this 
for the Java Broker Qmf plugin so the ObjectIds matched the assumptions 
made by qpidlibtools), if I don't set the ObjectId but do set idNames in 
the SchemaObjectClass then the objectName is formed from the 
concatenation of the values of the specified properties and the package 
and class name.

None of that is specified but seems to be consistent with what 
qpidlibtools has done.

The bottom line is that in many ways I think it should be opaque, but 
the ObjectId definitely has properties in the protocol and even the 
_object_name isn't opaque, though I definitely think that should be but 
I've implemented code that allows for the ambiguity of the specification 
- if nothing else is specified you'll see I default to a UUID.


>
>> And I presume that deleting and recreating queues can change their 
>> object
>> IDs?
>
> No, I believe deleting and recreating a queue will result in the same 
> identifier being used since the name is the same.

I'm not convinced that that statement is correct. I think that it is 
correct for the _object_name part of the ObjectId however as you'll note 
the _agent_epoch forms part of the ObjectId. Although it's optional I'm 
"pretty sure" that the broker passes the _agent_epoch value in QMF2 
ObjectIds and I'm also "pretty sure" that the C++ broker increments the 
epoch value each time it restarts.

I've not had time to check for certain, but I've got code in my Console 
class that says:


                                     // If there are any results 
available after evaluating the query we deliver them
                                     // via a SubscribeIndicationWorkItem.

                                     // Before we send the WorkItem we 
take a peek at the Agent Epoch value that forms
                                     // part of the ObjectID and compare 
it against the current Epoch value. If they
                                     // are different we send an 
AgentRestartedWorkItem. We *normally* check for Epoch
                                     // changes when we receive 
heartbeat indications, but unfortunately the broker
                                     // ManagementAgent pushes data 
*before* it pushes heartbeats. Its more useful
                                     // however for clients to know that 
an Agent has been restarted *before* they get
                                     // data from the restarted Agent 
(in case they need to reset any state).
                                     if (objectEpoch > agent.getEpoch())
                                     {
                                         agent.setEpoch(objectEpoch);
                                         agent.clearSchemaCache(); // 
Clear cache to force a lookup
                                         List<SchemaClassId> classes = 
getClasses(agent);
                                         getSchema(classes, agent); // 
Discover the schema for this Agent and cache it
                                         _log.info("Agent {} has been 
restarted", agentName);
                                         if (_discoverAgents && 
(_agentQuery == null || _agentQuery.evaluate(agent)))
                                         {
_eventListener.onEvent(new AgentRestartedWorkItem(agent));
                                         }
                                     }

                                     _eventListener.onEvent(
                                         new SubscriptionIndicationWorkItem(
                                             new 
SubscribeIndication(consoleHandle, resultList))
                                     );

It's been a while since I looked at that code, but I put it in place 
when I wrote the Java port of qpid-queue-stat which uses the QMF2 query 
subscription (I needed to emulate that on the client side by subscribing 
to the broker data push). As the comment says it's useful to be able to 
detect Agent restarts before the data from the restarted Agent gets sent 
(qpid-queue-stat resets its Maps when it gets an Agent restart event).

So I'm really pretty sure that ObjectIds will be different when a broker 
is restarted though I think that the _object_name part will be the same 
as that is formed from the package, class and name property (well for 
queues and exchanges it is for sure).

>
>> So, then, are Queue and Exchange the class (I'm guessing from the 
>> code I've
>> read).  Could you please explain packages?
>
> The package is really just a namespace. The goal of QMF was to be able 
> to define all different sorts of schemas and a namespace qualification 
> made this possible.
>
What Gordon says is pretty much it package is little more than a 
namespace used to help disambiguate in case the same class is seen in 
other Agents. It's worth looking through <qpid>/specs/management-schema.xml

Actually in the schema if you look at say Queue and Exchange you'll see:
     <property name="name"        type="sstr"  access="RC" index="y"/>

I suspect that the "index=" bit is suggesting that that particular 
property is being used to help form the _object_name, which ties in 
rather with the _id_names stuff I mentioned earlier.


I'm still not convinced it's a good idea for ObjectIds to be 
"interpretable/parsable" rather than opaque and as I say there's 
definitely something of an ambiguity between what is stated in the API 
document and what has been implemented in the qpidlibtools.


Not sure if any of the above actually helps :-) but hopefully it's 
useful. When I get a moment I'll add some debug to some test code to 
check on this, but I'm relatively sure what I've said is accurate from 
memory and given the comments and logic in my own code.

HTH
Frase


--------------040605000801050108000601--