Mailing-List: contact httpclient-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "HttpClient Project" <httpclient-dev@jakarta.apache.org>
Received-SPF: neutral (asf.osuosl.org: local policy)
Subject: Re: [HttpCore] NIO extensions: non-blocking client side transport
From: Oleg Kalnichevski <olegk@apache.org>
To: HttpClient Project <httpclient-dev@jakarta.apache.org>
In-Reply-To: <19196d860610091026o2acedfe4r5dc4f533b5ac7e0e@mail.gmail.com>
References: <1160062264.4939.5.camel@localhost.localdomain>
	 <452677D4.8020300@dubioso.net>
	 <1160157101.5324.43.camel@localhost.localdomain>
	 <4527D46C.1090709@dubioso.net>
	 <19196d860610070933i4e06b97er8138ff6d02cb6918@mail.gmail.com>
	 <1160324107.4899.23.camel@localhost.localdomain>
	 <452A6BF3.10806@khelekore.org>
	 <19196d860610091026o2acedfe4r5dc4f533b5ac7e0e@mail.gmail.com>
Content-Type: text/plain
Date: Mon, 09 Oct 2006 20:28:03 +0200
Message-Id: <1160418483.4972.21.camel@localhost.localdomain>
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

On Mon, 2006-10-09 at 13:26 -0400, Sam Berlin wrote:
> I agree with Robert that it is much easier to go from an event driven
> model to a blocking model.  If the first layer that HttpNIO exposes is
> blocking, there'd need to be additional hacking below that in order to
> remove the blocking / thread-based layer.  On the other hand, if the
> first layer it exposes is non-blocking, it's relatively trivial to add
> a thread ontop of that and expose an additional blocking layer.
> 
> It is difficult to think of many scenarios that require (or a better
> with) non-blocking I/O, but I would caution against excluding them
> from HttpClient's scope.  If HttpClient 4.0 had been ready a year or
> so ago (with an exposed non-blocking layer), we would definitely have
> used it in LimeWire as the basis of file-transfers.  As-is, we
> invented our own minimalistic non-blocking state-based http transport
> for downloads.
> 
> If the non-blocking layer is there, I guarantee that folks will be
> able to find a use for it.  Whereas if only a blocking layer is there,
> those developers looking for the high-performance asyncronous model
> will have to go elsewhere.
> 

Sam, Robert, et al

Simply for practical reasons while there are only two guys hacking (me
and Roland) we ought not spread out efforts too thin. A full-blown
even-driven API will take time to get right. I think it is more
important to get HttpCore ALPHA3 release that covers 95% of use cases
out the door rather sooner than later. Beyond that it is just a matter
of priorities and available time.  

Oleg


> Sam
> 
> On 10/9/06, Robert Olofsson <robo@khelekore.org> wrote:
> >
> > Since my proxy is almost fully nio/event based I would like to share
> > a few comments.
> >
> > Oleg Kalnichevski wrote:
> > > I am still quite skeptical about usefulness of a fully event-driven HTTP
> > > transport for one simple reason: asynchronous (non-blocking) I/O
> > > transport makes no sense of what so ever if the process of content
> > > generation or content consumption is asynchronous (blocking).
> >
> > There are many things that may block here are a few examples:
> > *) DNS-lookup
> > *) File reading writing
> > *) Database access
> > *) All higher level api:s that only give you a stream.
> > *) Calls to Runtime.exec
> >
> > That DNS lookups are also totally single threaded in native code in
> > some systems does not make things better.
> >
> > > If one
> > > needs a worker thread to generate / process content anyways, what is the
> > > point of having an even driven transport?
> >
> > Agreed.
> > One objection here may be that you do not need one worker thread for all
> > of the content generation, but that usually does not make things better.
> > You will still need the worker thread for the _slow_and_blocking_
> > operation.
> > So if content generation/modification uses any of the above then using
> > workers simplify things a lot.
> >
> > > I see only a few scenarios
> > > where the third choice (event callbacks) may prove advantageous,
> > > primarily in HTTP proxies and gateways.
> >
> > Except that http proxies does lots of dns lookups so they will block
> > a lot. My proxy spawns worker thread only when they need to, but it
> > complicates some part of the code.
> >
> > That my proxy also modifies the content and caches the data will mean
> > lots of other blocking calls in some of the code paths.
> >
> > > I think ultimately we need both options. I suggest we start with the
> > > second option, release ALPHA3 and then consider implementing the third
> > > option before ALPHA4 / BETA1.
> >
> > One thing to keep in mind:
> > It is easier to go from event driven to a blocking model than to do the
> > reverse. This may be an argument to go for number 3 (full nio).
> > If you go for 3 then make it easy to use a few selector-threads
> > otherwise the system will use only 1 cpu (or 1 core).
> >
> > /robo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org