qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Ross <tr...@redhat.com>
Subject Re: Technical Report on Qpid Broker Federation Experiments
Date Fri, 29 May 2009 13:09:21 GMT
Hi Greg,

Thanks for sharing this report.  Your detailed results are interesting, 
especially since this is a level of benchmarking that we haven't gotten 
to ourselves.

I'll take a whack at some of your questions in-line below.  Hopefully 
others will chime in as well.

First or all, I'm curious as to how you configured your federation 
routes.  The paper details the physical broker topologies but doesn't 
mention which qpid-route options were used to set up your test 
networks.  This, of course, has performance implications.



gregory james marsh wrote:
> Hi All,
> A few other lab members and myself have just completed an Ohio State 
> University Technical Report dealing with Qpid broker federation 
> entitled "Scaling Advanced Message Queuing Protocol (AMQP) 
> Architecture with Broker Federation and InfiniBand".  The pdf is 
> publicly available at http://nowlab.cse.ohio-state.edu/projects/amqp/ 
> as well as ftp://ftp.cse.ohio-state.edu/pub/tech-report/TRList.html
> Basically this report tests federation with k-nomial tree topology.  
> We were unable to use Qpid rdma/InfiniBand (IB) due to problem 
> documented on JIRA (QPID-1855).  So we performed our experiments using 
> IPoIB via Qpid TCP connector.
> Based on what we saw in our experiments, each of us has written some 
> questions for the list about how federation is designed & implemented 
> internally.  I've compiled the questions below.  I make reference to 
> figures and tables from the report where appropriate.
> 1. In detail how does the connection of a destination broker to a 
> source broker with "qpid-route route add" differ from a consumer 
> application declaring and binding a queue on a source broker?  Are 
> there functional differences in how messages are routed to a federated 
> destination broker as opposed to a consumer?  What anticipated affect 
> would these differences have on host system performance?
In case you haven't already seen it, there's a Wiki page on federation 
at http://qpid.apache.org/using-broker-federation.html.

In Qpid federation, the destination broker behaves as a normal AMQP 
client to the source broker.  With "qpid-route route add" and 
"qpid-route dynamic add", the destination broker declares a private, 
auto-delete queue on the source broker and creates a subscription to 
that queue.  The "route add" command causes the queue to be bound 
explicitly to an exchange on the source broker.  The "dynamic add" does 
not directly bind the queue but dynamically creates and deletes bindings 
as consumer applications add bindings to the destination exchange.

Federation does not introduce any differences in the way messages are 
forwarded.  It simply uses the normal AMQP mechanisms.  The "internal" 
federation client exposes less capability than is offered by the Qpid 
client APIs.  In particular, delivery policy and flow control.  These 
are issues that will be addressed in future development.
> 2. Why does a chain of federated brokers to one consumer have a higher 
> bytes/sec rate on their interfaces (using SYSSTAT/sar to collect 
> traffic data on IPoIB interface) than a single broker to one 
> consumer?  This experiment is explained in Sections 3.1, 4.2, Figure 8a.
There are lots of variables here.  The text and diagrams in the paper 
don't provide enough detail to definitively answer this question.  It's 
possible that a different delivery policy on your clients (vs. the 
federation links) is a cause of the disparity.  In M4, the federation 
links are synchronous (i.e. ack per message).  In 0.5 (the next 
release), there's an option for batching acknowledgements on federation 

If your test clients use more efficient asynchronous transfers, the 
federation links will have higher acknowledgement traffic than will the 
normal client links (in M4).
> 3. Why does the activity at the root broker and producer increase as 
> we increase the federation tree?  Our understanding was that the only 
> work root broker/producer does is send messages to broker(s) they are 
> directly connected to.  (This is similar to question 2, but different 
> experiment. In such situations the net interfaces on the producers and 
> consumers (as well as brokers) of more complex trees showed elevatated 
> bytes/sec rates compared to less complex trees.  Experiment is 
> explained in Sections 3.3, 4.4, Figure 14a.)
I'd be interested to see exactly how you configured your federation 
routes in this scenario.
> 4.  We observed that the tree P-B-4B-16C was performing better than 
> the tree P-B-4B-4C. And I thought this was because more consumers 
> being attached keeps the leaf brokers busy thus reducing the 
> contention at the root broker. Can we confirm if this understanding is 
> correct.  (This is a comparison of the experiments performed in 
> Sections 3.3/4.4 and 5. Compare numbers in Tables 5, 6, 7, 8, 
> particularly the small message rate numbers in Tables 5 & 8 for 64B 
> and 256B.)
> 5.  What is the advised measurement for performance in federated 
> broker-consumer design?
> (We used the benchmarks we created for our SC08 workshop paper last 
> fall--also available at above sites.  Delivered bandwidth in the 
> current report is our benchmark bandwidth multiplied by number of 
> consumers).
> 6.  Which of the tests included with Qpid release might help us get 
> more insights on our topology study?
> Thanks in advance for an comments and insights.
> Greg Marsh
> Network Based Computing Lab
> Ohio State University
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:users-subscribe@qpid.apache.org

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

View raw message