qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Schloming <...@alum.mit.edu>
Subject Re: Proton's new async features - any examples? 'cause I'm scratching my head!
Date Sat, 15 Mar 2014 13:15:56 GMT
On Sat, Mar 15, 2014 at 5:38 AM, Fraser Adams <fraser.adams@blueyonder.co.uk
> wrote:

> Thanks Rafael,
> I'll take a look at this.
> TBH I had actually done a bit of grepping around and had come across
>  SelectableMessengerTest, but unfortunately I think that only confused me
> more :-/
> I guess that my mental model of how I'd expect to use this would be in a
> notifier loop analogous to a "broken apart" version of:
>   while (1) {
>     pn_messenger_work(messenger, -1); // Block indefinitely until there
> has been socket activity.
>     process();
>   }
> So seeing the non-blocking select and the way Pump was being used in a
> "busy wait" loop messed with my head - though I suspect that was just down
> to me "over analysing" things :-D

Ah, yes. The "busy wait" as you call it is actually just finishing up all
pending I/O operations. The loop terminates as soon as there is no more I/O
to perform. This works because the test is sequentially scripting both
sides of the socket interaction, so there is never any need to wait around
for a concurrent thread/process to finish sending you data, and therefore
no point in having a timeout that is > 0. If the body of pump_once were to
use a non zero timeout and be put in an infinite loop then you basically
have a "normal" event loop (minus the deadline part).

> I have to say that I agree with the comments raised by Darryl Pierce
> yesterday evening about some of the names being a little confusing.
> Out of curiosity how much more efficient would you expect this approach to
> be? I'm guessing if you have several messenger instances on the go and a
> high message rate it's likely to make quite a difference and you can also
> use epoll now too I guess.

I actually wouldn't expect it to make all that much of a difference
performance-wise since messenger's internal wait loop is implemented using
the same selectables API that it exposes. In fact for a language like
python it might be marginally faster to leave the loop in C rather than
reimplementing it in python. The API is really more about flexibility with
how you embed messenger into your environment. If you have an existing I/O
loop then this allows you to use it. The only case I can think of where
this would make a difference performance-wise is if you wanted to have
1000's of messengers in a single process without having a thread per

As for select vs poll vs epoll, the internal wait loop uses poll, however
the performance difference between select, poll, and epoll really depends
not only on the total number of connections you have, but on how your I/O
activity is distributed. The case where epoll really shines is if you have
a lot of idle connections and a few active ones. This is because the poll
interface forces you to loop over all the fds to figure out which ones have
events, whereas epoll just gives you back the events directly. If a high
proportion of your fds frequently have events though, then epoll's
advantage diminishes. In fact I've seen comparisons where poll comes out
faster between the two.

> FYI when I tried "make docs" it mostly looks like it works but I saw
> several warnings:
> warning: ignoring unsupported tag `INLINE_SIMPLE_STRUCTS  =' at line 313,
> file user.doxygen
> warning: ignoring unsupported tag `CITE_BIB_FILES         =' at line 583,
> file user.doxygen
> warning: ignoring unsupported tag `MATHJAX_EXTENSIONS     =' at line 1166,
> file user.doxygen
> warning: ignoring unsupported tag `LATEX_BIB_STYLE        =' at line 1285,
> file user.doxygen
> warning: ignoring unsupported tag `INTERACTIVE_SVG        =' at line 1699,
> file user.doxygen
Hmm, I don't seem to see these. What version of doxygen are you using?

> and
> Scanning dependencies of target docs-py
> +-----------------------------------------------------------
> ---------------
> | In /home/fadams/qpid/qpid-trunk/proton/proton-c/bindings/python/
> | proton.py:
> | Import failed (but source code parsing was successful).
> |     Error: NameError: name 'PN_CONNECTION_STATE' is not defined (line
> |            3292)
> |
> It looks like your new API docs are created OK, but I've raised PROTON-535
> to cover it.

Did you do a make prior to doing make docs or just do make docs? The epydoc
generation imports the swig modules so it needs the C code to be built in
order to run in its entirety, however it falls back to source only mode if
the import fails.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message