qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Freeman <ke1g...@gmail.com>
Subject Re: Questions from a novice
Date Mon, 01 Apr 2013 21:15:12 GMT

Thanks again.

There are a few comments interspersed below, but no reply is required.

On Mon, Apr 1, 2013 at 3:48 PM, Fraser Adams

> On 1 Apr 2013, at 19:56, Bill Freeman <ke1g.nh@gmail.com> wrote:
> >
> >
> > I believe that I fall into the data push camp, even though I may not be
> > using the modern means of achieving it.  I create my qmf.console.Session
> > instance with rcvObjects = True.  Then, after an initial one copy of
> > everything, I'm only sent updates if anything changes, and then at most
> > once every 10 seconds.  I don't want to poll.  So maybe the flag works
> > (though I have other concerns about a broker wide, rather than connection
> > or even session specific setting).
> >
> That's definitely a QMF1 API that you are using then, there are actually
> two python QMF1 APIs one uses getObjects() calls and the other has a
> notification type API. The 10 second interval can be charged by a broker
> switch BTW I can't recall exactly what it is off the top of my head, but if
> you do qpidd -h it should hopefully be obvious.

I'm actually pretty happy with 10 seconds, but I did see where to set this
in the qpidd arguments/configuration.  I don't remember it either, but I
know where to look it up.  It's another one of those broker wide settings
that I'd rather not enforce on other tools connecting to these brokers (if
there are any).

And yes, I agree, moving to QMF2, and later an AMQP standard, seem
desirable.  But both of those steps will be for a future deliverable for me.

> As an aside you mentioned previously about cumin "missing" updates,
> possibly that's true, but it shouldn't really matter and your application
> should probably be designed to be tolerant of this sort of thing - the
> updates of things like queueDepth, msgEnqueue, msgDequeue etc. are all
> "incremental" so there's no concept of "resetting" values a client has to
> track an initial value and deltas, it's worth pointing out though that
> timestamps are sent with each update, so if an update has been delayed in
> transit you can use the update time stamp to know the timestamp from the
> broker's perspective.

I realize that the final numbers come out the same.  But this is to be an
interactive tool, used by people who are panicking because some queue seems
wedged.  So if they command a change, I'd really like to show them a
result.  Within 10 seconds seems fine.  Not until the next time something
happens to the queue probably is not.

> >
> >
> > Whatever its called, I need some kind of key to disambiguate objects.  I
> > need them to be consistent across restarts of the broker, and maybe even
> > across some minor reconfiguration of the broker that is outside of my
> > control (such as the deletion and/or addition of queues, exchanges, and
> > bindings).  Since I will be talking to multiple brokers, I need to guard
> > against the possibility that a key could be repeated in a second broker
> > (particularly when multiple brokers are federated to duplicate efforts
> as a
> > fallback if one crashes).
> It has been a while since I looked at the guts of QMF2 ObjectIds as I say
> I tend to treat them as opaque. I think that Gordon's previous comments
> about V2 OIDs looked about right, but I'd need to have a proper look. As he
> said V1 and V2 OIDs are rather different.
> In V2 there's an epoch value that changes across broker restarts, I think
> that only applies to the Agent name and not the OIDs, but it has been a
> while, I'll have a proper look when I get IT back if you can bear with me.
> You mention being consistent across broker restarts, I'm assuming you are
> talking about persistent queues? Clearly non-persistent queues will be
> different objects. I'm not sure whether the broker treats recreated
> persistent queues as the same objects each restart, possibly but I can't be
> sure, it might not even be the right thing to do that, if you did the same
> thing with say an in memory object reference they would definitely be at
> different addresses and TBH that's usually how I view ObjectIds - as a way
> of describing an address at a point in time. I'll need to take a prober
> look to ascertain the current truth though.

I personally don't care whether I use object IDs or not.  In fact, I'd
rather use the queue name or exchange name, etc.  That's the real problem
for me with the V1 interface.  I get two separate callbacks.  One has
properties (and I think that I only get this once per object, and I've only
looked at queues so far), and thus has the name.  The other callback has
statistics, but no properties, so I can't use the name property to look up
the (local cached) object for which this is a statistics update.  So I'm
using str(xxx.getObjectId()) to match them up*.

[* I think that some of the code in qmf.console has some dictionaries
indexed by ObjectId instances themselves, rather than their string values,
which I'm not sure is wise.  But I'd have to go back and look again to be
sure. ]

> >
> >
> > I'm much better at Python than I am at Java, and my web server (which
> > doesn't speak QMF anyway, being isolated by REDIS) is tornado, a python
> > based tool.
> Fair enough if you are more familiar with python, but I really really
> would recommend moving away from QMF1 and perhaps use the lib tool API
> Gordon mentioned earlier. As I mentioned before management is likely to be
> in for something of a reshuffle, I think it's likely that there will be
> attempts to make transitioning from QMF2 as easy as possible, but that
> probably won't hold for QMF1.

And I'll follow along as I can.

> > And Rest doesn't sound promising for getting updates pushed to
> > me.
> >
> So you may be right, though QMF2 has a getWorkItem() call which can be
> blocking, that is used in QMF2 in place of a pure asynchronous model as
> there was talk in the spec of avoiding threading issues. I implemented this
> in REST via a long-polling call which works pretty well, my GUI actually
> works by getting notified on heartbeats and other asynchronous events such
> as queueDeclare etc. when triggered it then does a synchronous getObjects()
> call - via REST which ultimately calls my Java getObjects(). I won't get
> into the relative pros and cons of long-polling versus WebSockets, I
> generally agree that the latter is the way to go, but I needed wide browser
> support and TBH I need to get a bit more up to speed on WebSockets, it's
> definitely on my list of things to do, but the current approach works
> pretty well, and is arguably a pretty good mapping to the underlying API
> call.

We python folks have a particularly high motivation to avoid threads.
Tornado's event loop model (something like Twisted, but different) is
pretty slick on OSes that support epoll or select.

WebSockets is pretty nifty.  Comet done right.  Pretty easy to use.  An
awful lot of browsers support it now.  There is a JavaScript library,
socksjs (apparently more than one: socket.io), that will use it if it's
there and emulate it if it's not, using long poll or whatever old Comet
technology works, even being able to fall back to polling with GETs.  Of
course you have to code your server to handle the GETs if you want to fall
back that far.  That's not to say that I've used it.  Plain WebSockets has
been working in my test environment: Chrome, recent Firefoxes, even the
most recent IEs.

> >
> >> One last thing, there is some discussion beginning about the future of
> >> management, the eventual intention is to move away from QMF towards a
> new
> >> AMQP management API, however that is in the very early stages so at the
> >> moment QMF is the only game in town for C++ broker management. There
> are a
> >> few of us very keen to make sure that transition to the new API is as
> >> painless as possible, but as I say it's very very early days and is
> mainly
> >> said to try to ensure that you bear it in mind with your application.
> I'd
> >> very definitely avoid QMF1 though QMF2 is a better bet.
> >>
> >
> >
> > And my target brokers, which I don't get to specify, are C++ brokers.
> These support QMF1 and QMF2, but from a client perspective I'd really
> recommend avoiding QMF1 if you possibly can.

As soon as I understand how not to poll...

> The data push stuff is supported in both QMF1 and QMF2 formats. The QMF2
> API allows a type of query subscription, but the C++ broker doesn't
> actually support this :-( I actually managed to fake this in my Java
> implementation by doing a data subscription and filtering on the client
> side to implement the query API. That said it might have been overkill, I
> went to town implementing the full QMF2 API, but it hasn't really taken off
> and the python lib tools stuff, although it implements the QMF2 protocol
> doesn't really implement a documented API.

By which you mean documented in the XML?  Or is there a documentation link
that I haven't found?  (I've read so much stuff over the last couple of
months that my head is still spinning pretty faxt.)

> The lack of standardisation of QMF2 and the hitherto lack of support on
> the Java broker is one of the things driving the moves to try and create a
> properly standardised AMQP 1.0 management protocol and API.
> I've been working on QMF2 support for the Java broker and it's something
> of a mission of mine to try and improve the overall cohesion of AMQP
> management, that was one of the key drivers behind the GUI I put together
> and the work I'm starting on the Java broker.

Cohesion would be great.

> I'll try to give a bit more detail later in the week on some of this, some
> of it will require me to have a bit of an eyeball at some code.
> No rush.  I'll have already done whatever I'm going to do for tomorrow's
dog and pony show.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message