qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Stitcher <astitc...@redhat.com>
Subject Re: Rdma IO state transitions [Was: AsynchIO state transition]
Date Wed, 15 Sep 2010 16:04:20 GMT
On Tue, 2010-09-14 at 12:39 -0700, a fabbri wrote:
> Thanks for the comments Andrew... this thread piqued my interest.. see
> inline comments below.
> On Tue, Sep 14, 2010 at 7:47 AM, Andrew Stitcher <astitcher@redhat.com> wrote:
> > >From the description it seems that you are asking about the RDMA
> > AsynchIO state transition. You should have said that as part of the
> > description as it is entirely different from and other part of the IO
> > system of qpid.
> Having multiple AsynchIO classes is confusing, IMHO, even though they
> are in separate namespaces.

I agree, but when adding the rdma implementation I wasn't able to
refactor the various classes into single classes so I kept the same
overall design for all protocols. In the near future I'd very much like
to refactor all of this code so that all the commonality is in the same

> >
> > On Mon, 2010-09-13 at 15:04 -0700, Danny Guo wrote:
> >> Hi all,
> >>
> >>
> >> Is there a simplistic way to understand the state transition? I got confused
> >> with the oldState, newState and “action”.
> >
> > I find that drawing a state diagram helps me to understand what's going
> > on - I find I need to do this about every 6 months (and I wrote the code
> > in question).
> I also found this code difficult to read and understand.  It is not
> the idea of spinning on an atomic interlock (state) variable that is
> confusing to me, but digesting the state semantics and actions you
> take based on transitions.
> Maybe someone should contribute a dot (graphviz) file of the state
> transitions and check it in wherever design docs go.

As long as they maintain it too :-)

> We could just use pthread_spin_lock() around your state transition
> case statement, instead of spinning on the primitive compare and swap.
>  Not sure how much more readable that would be, but it happens to be
> more my style.  ;-)

It's not clear to me this would necessarily avoid the contention, but it
is certainly worth thinking about.

> <snip>
> > The entire purpose of this state machine is to arbitrate between calls
> > to dataEvent() and notifyPendingWrite() which can happen on different
> > threads at the same time.
> Segue to a related question I have... Can you help me understand, or
> just point to docs/code, the threads involved here?
> The upper layers ("app") call notifyPendingWrite() from whatever
> thread they want.  dataEvent() gets called from the poller thread.  Is
> it correct that there is typically only one poller thread per
> Blah::AsynchIO instance?

No, the IO threads are entirely independent of the number of
connections. The rule is something like 1 IO thread per CPU (this needs
to be revisited in the light of the NUMA nature of current multicore,
multisocket machines).

The IO threads all loop in parallel doing something like:

  Wait for IO work to do
  Do it.

The upper layer threads can be entirely different threads (in the case
of a client) or in the case of the broker an arbitrary IO thread
(possibly one currently processing an event from this connection,
possibly processing another connection). The broker does nearly all its
work on IO threads. The exceptions are Timer events which are at least
initiated on their own thread, and I think some management related work.

> Where does the --worker-threads=N arg to the CPP qpidd broker come into play?

This overrides the default selection of number of IO threads.

> Finally--perhaps a can of worms-- but why does notifyPendingWrite()
> exist, instead of just writeThis().  Is this part of the "bottom-up
> IO" design?  I feel like having the app tell us it wants to write (so
> call me back) is more complex than just having a writeThis(buf)
> method.

It currently works like this to ensure that the actual writes happen
correctly serialised to the connection processing, ie when the callback
for "please write something" happens we can be sure that nothing else is
happening on the connection.


Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

View raw message