hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sammer <e...@lifeless.net>
Subject Re: HTTP transport?
Date Tue, 06 Oct 2009 03:15:08 GMT
Scott Carey wrote:
> Even in the beacon case, if the browser is likely to send another request
> shortly, it cuts the effective network latency in half.

Which is generally not the case in the beacon / ad server use case. That
was the only point I was making. That's besides the point, though. I
think we both agree that KA for something like Avro transport is
probably good.

> Establishing a TCP
> connection is at minimum one full round trip -- before the request.  If
> latency is important KeepAlive is useful as long as a second request is
> expected in a short enough time.
> As long as the server is not process or thread per connection, one can scale
> up connection count rather high (20k) if necessary.
> With respect to Avro/Hadoop, I suspect requests from clients to be time
> clustered.

That was my thought as well. The thing that gets me is that in the case
of Hadoop (and the related subprojects) the clients utilizing this
particular HTTP connection are probably going to be pretty small (maybe
low thousands?). This is even better for keep alive as there's a solid
chance you're going to have a high reuse rate. Of course, I'm assuming
we're talking about things like name node to data node, hbase client to
region servers, and those types of communications. Even if you just used
Jetty (or any other thin HTTP 1.1 container that supports KA), one
should easily be able to see good performance.

Eric Sammer

View raw message