Mailing-List: contact dev-help@hc.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpComponents Project" <dev@hc.apache.org>
Received-SPF: pass (athena.apache.org: domain of sberlin@gmail.com designates
 72.14.204.232 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=C0HS+pHGJLXbJVvg3OYJWK3Eej8tXoxuzpHsVZy3It+nAkgRy6jtGL6R5p7TONuOybPVkHFtenL4XBuqPF0Z5buYDyDP1CelxcfCAQwryPz3/7e5EAS8TjTH9J9lyyqs3xcUg0ubwiWynlfhuZ+7PZ30EWlHTd5LY9j8EdP8Dg0=
Message-ID: <19196d860801231017n15a0c76awc8bf5bbdbcf5db65@mail.gmail.com>
Date: Wed, 23 Jan 2008 13:17:55 -0500
From: "Sam Berlin" <sberlin@gmail.com>
To: "HttpComponents Project" <dev@hc.apache.org>
Subject: Re: async http clients
In-Reply-To: <1201083283.5675.17.camel@ubuntu>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <3735543.1201038759822.JavaMail.root@wombat.diezmil.com>
	 <1201040578.5742.13.camel@ubuntu>
	 <43F92612-3A20-4090-A2E4-107E6B1362B2@mac.com>
	 <1201083283.5675.17.camel@ubuntu>

There were definitely bugs with AbstractSelector & SelectionKeys that
could cause deadlocks to happen in rare scenarios.  See
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4744057 for one.
Newer versions of the JDK have fixed this.  There were also problems
on OS X if using the default selector (kqueue) that caused the entire
system to hang.  I'm not positive if newer releases have fixed this,
as we switched to not using kqueue (-Djava.nio.preferSelect=true).

Sam

On 1/23/08, Oleg Kalnichevski <olegk@apache.org> wrote:
>
> On Tue, 2008-01-22 at 22:23 -0800, David Koski wrote:
> > Hi,
> >
> > Sadly, I can't claim I have personal knowledge of this being true.  I
> > have an application where I can get a reactor thread to hang like that
> > and this description seemed to fit.  I have been unable to make a unit
> > test that reproduces this behavior however.
> >
> > I am using:
> >
> > java version "1.5.0_06"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-113)
> > Java HotSpot(TM) Server VM (build 1.5.0_06-68, mixed mode)
> >
> > Here is another reference:
> >
> >       http://forum.java.sun.com/thread.jspa?threadID=459338&start=120
> >
> > "register and interestOps will block if they are done from another
> > thread while the select is active. That's why I use a list of things
> > that must be done in the select loop."
> >
> > And I read this:
> >
> >       http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/SelectionKey.html
> >
> > Selection keys are safe for use by multiple concurrent threads. **The
> > operations of reading and writing the interest set will, in general,
> > be synchronized with certain operations of the selector. Exactly how
> > this synchronization is performed is implementation-dependent: In a
> > naive implementation, reading or writing the interest set may block
> > indefinitely if a selection operation is already in progress**; in a
> > high-performance implementation, reading or writing the interest set
> > may block briefly, if at all. In any case, a selection operation will
> > always use the interest-set value that was current at the moment that
> > the operation began.
> >
>
> The key here is the word 'naive'. I have a hard time believing Sun chose
> a naive implementation for its JREs. Inability to modify interest
> operations on the selection key from a different thread _utterly_
> defeats its purpose.
>
> NIO used to be (and still is) quite buggy. So, before I start working on
> an ugly workaround I would like to be reasonably sure this is not a bug
> in Sun's older JREs. I would also like to know to what extent different
> platforms are affected. The JRE you are using is fairly out of date.
> Could you please upgrade to the latest Java 1.5 release (1.5.0_14) and
> see if the problem is still reproducible? Have you tried Java 1.6? What
> OS are you on?
>
> Oleg
>
> > To me, that suggests that select + modify of the interest ops == bad,
> > which agrees with what the various posts say.  Again, I have no nio
> > expertise, so my idea may be bunk.
> >
> > David Koski
> >
> >
> > On Jan 22, 2008, at 2:22 PM, Oleg Kalnichevski wrote:
> >
> > >
> > > On Tue, 2008-01-22 at 16:52 -0500, david_koski@mac.com wrote:
> > >> Hi,
> > >>
> > >> Your suggestion worked well, but I think I ran into a problem with
> > >> it (using A6, if it matters).  I ended up with a hung IOReactor
> > >> thread:
> > >>
> > >> Thread [I/O dispatcher 2] (Suspended)
> > >>    KQueueArrayWrapper.register0(int, int, int, int) line: not
> > >> available [native method]
> > >>    KQueueArrayWrapper.setInterest(int, int) line: 99
> > >>    KQueueSelectorImpl.putEventOps(SelectionKeyImpl, int) line: 179
> > >>    SocketChannelImpl.translateAndSetInterestOps(int,
> > >> SelectionKeyImpl) line: 733
> > >>    SelectionKeyImpl.nioInterestOps(int) line: 87
> > >>    SelectionKeyImpl.interestOps(int) line: 65
> > >>    IOSessionImpl.clearEvent(int) line: 125
> > >>    AsyncHTTPClient
> > >> $
> > >> AsyncConnection
> > >> (DefaultNHttpClientConnection).produceOutput(NHttpClientHandler)
> > >> line: 183
> > >>    AsyncHTTPClient
> > >> $EventDispatch(DefaultClientIOEventDispatch).outputReady(IOSession)
> > >> line: 102
> > >>    AsyncHTTPClient$EventDispatch.outputReady(IOSession) line: 353
> > >>    BaseIOReactor.writable(SelectionKey) line: 109
> > >>    BaseIOReactor(AbstractIOReactor).processEvent(SelectionKey) line:
> > >> 192
> > >>    BaseIOReactor(AbstractIOReactor).processEvents(Set) line: 174
> > >>    BaseIOReactor(AbstractIOReactor).execute() line: 137
> > >>    BaseIOReactor.execute(IOEventDispatch) line: 69
> > >>    AbstractMultiworkerIOReactor$Worker.run() line: 281
> > >>    Thread.run() line: 613
> > >>
> > >> The AsyncHTTPClient class is mine, but it is just very thin
> > >> wrappers on the default implementations.  Anyway, the thread hangs
> > >> like this while trying to clear the write interest on the
> > >> SelectionKey.  I only seem to run into this at high (> 1000 per
> > >> second) request rates.
> > >>
> > >> I think this may actually be related to waking up the the IOControl:
> > >>
> > >>                conn.requestOutput();
> > >>
> > >> this ends up here:
> > >>
> > >>
> > >>    public void setEvent(int op) {
> > >>        if (this.status == CLOSED) {
> > >>            return;
> > >>        }
> > >>        synchronized (this.key) {
> > >>            int ops = this.key.interestOps();
> > >>            this.key.interestOps(ops | op);
> > >>            this.key.selector().wakeup();
> > >>        }
> > >>    }
> > >>
> > >> However, several sites suggest that concurrent modification of
> > >> SelectionKeys is a recipe for disaster:
> > >>
> > >> http://rox-xmlrpc.sourceforge.net/niotut/index.html
> > >>
> > >> As a result, if you plan to hang onto your sanity don't modify the
> > >> selector from any thread other than the selecting thread. This
> > >> includes modifying the interest ops set for a selection key,
> > >> registering new channels with the selector, and cancelling existing
> > >> channels.
> > >>
> > >>
> > >>
> > >> But that is _exactly_ what I am doing here.  The thread that wants
> > >> the connection is touching the SelectionKey and the IOReactor (I/O
> > >> dispatcher) thread is also touching it.
> > >>
> > >> Am I understanding this correctly?
> > >>
> > >
> > > David,
> > >
> > > The javadocs of the SelectonKey clearly states the class is threading
> > > safe [1]:
> > >
> > > "... Selection keys are safe for use by multiple concurrent
> > > threads..."
> > >
> > > One certainly should not attempt to access the selector from multiple
> > > threads but it is completely new to me some people think this also
> > > applies to the selection keys. Selection keys would be completely
> > > pointless if they were not threading safe
> > >
> > > Can this be a JRE issue? What is the JRE you are using?
> > >
> > > Oleg
> > >
> > > [1]
> > > http://java.sun.com/j2se/1.4.2/docs/api/java/nio/channels/SelectionKey.html
> > >
> > >
> > >> Thanks,
> > >> David Koski
> > >>
> > >>
> > >> -- Oleg Kalnichevski wrote :
> > >> On Thu, 2007-07-12 at 14:10 -0700, David Koski wrote:
> > >>> Hi,
> > >>>
> > >>> I have been reading NHttpClient and I think I finally understand how
> > >>> it all works, but I am a bit stuck with how to use it.  Let's say I
> > >>> wanted to build something along the lines of a load balancer:  many
> > >>> incoming connections, many outgoing connections, most are idle or
> > >>> waiting for a response.
> > >>>
> > >>> Focusing on the outgoing connections part, I want to have keep-alive
> > >>> connections to a set of hosts, multiple connections per port.  For
> > >>> example:
> > >>>
> > >>>   LB -> host1:80
> > >>>   LB -> host1:80
> > >>>   LB -> host1:80
> > >>>
> > >>>   LB -> host2:80
> > >>>   LB -> host2:80
> > >>>
> > >>> If I have a queue of operations I want to do, I can see how I might
> > >>> use a series of ioReactor.connect() calls to create the connections
> > >>> and have submitRequest() methods in my HttpRequestExecutionHandler
> > >>> pull them off the queue and service them.
> > >>>
> > >>> The problem I am having is dealing with the steady state.  I would
> > >>> like to keep these connections around for a while (and indeed using
> > >>> the DefaultConnectionReuseStrategy they are kept alive.  However,
> > >>> once
> > >>> my queue drains and my submitRequest() method returns null, how do I
> > >>> wake the handlers back up?  I can see any way to get the reactor to
> > >>> call back into my handler without opening a new connection.
> > >>>
> > >>> Am I missing something?  Or going about this the wrong way?
> > >>>
> > >>
> > >> Hi David
> > >>
> > >> Just invoke IOControl#requestOutput() (implemented by all NHttp
> > >> connections) and it will cause the I/O reactor to fire up the
> > >> NHttpClientHandler#requestReady() event, which you can use to
> > >> submit a
> > >> new request on that connection
> > >>
> > >> Hope this helps
> > >>
> > >> Oleg
> > >>
> > >>> Thanks,
> > >>> David Koski
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jaka...
> > >>> For additional commands, e-mail: httpcomponents-dev-help@jaka...
> > >>>
> > >>>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: httpcomponents-dev-unsubscribe@jaka...
> > >> For additional commands, e-mail: httpcomponents-dev-help@jaka...
> > >>
> > >>
> > >> --
> > >> This message was sent on behalf of david_koski@mac.com at
> > >> openSubscriber.com
> > >> http://www.opensubscriber.com/message/httpcomponents-dev@jakarta.apache.org/7128633.html
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> > >> For additional commands, e-mail: dev-help@hc.apache.org
> > >>
> > >>
> > >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org