hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: HTTP transport?
Date Tue, 06 Oct 2009 02:59:24 GMT

On 10/5/09 1:53 PM, "Eric Sammer" <eric@lifeless.net> wrote:

> Ryan:
> Certainly keep alive will help in this case, if that's what you're
> referring to. The server holds the socket for N seconds or M requests,
> which ever comes first. What you're saving with KA is the connection
> setup / tear down. If you have a lot of cases where the client makes a
> single request and goes away, then KA hurts because the server holds the
> connection for the KA timeout (N seconds). This *really* helps if you're
> using TLS due to the additional connection setup overhead.
> It's my opinion and experience that KA helps greatly in the case of many
> exchanges between a small to medium number of clients and a server such
> as RPC. The anti-example is an ad server or web beacon server, for instance.

Even in the beacon case, if the browser is likely to send another request
shortly, it cuts the effective network latency in half.  Establishing a TCP
connection is at minimum one full round trip -- before the request.  If
latency is important KeepAlive is useful as long as a second request is
expected in a short enough time.

As long as the server is not process or thread per connection, one can scale
up connection count rather high (20k) if necessary.

With respect to Avro/Hadoop, I suspect requests from clients to be time

> Regards.
> Ryan Rawson wrote:
>> I have a question about these headers... will they impact the ability to do
>> many, but small, rpcs? Imagine you'd need to support 5,000 to 50,000
>> rpcs/second. Would this help or hinder?
>> On Oct 5, 2009 4:44 PM, "Eric Sammer" <eric@lifeless.net> wrote:
>> Doug Cutting wrote: > More or less. Except we can probably arrange to omit
>> most of those > response...
>> Content-Type and Server are probably unavoidable. Some of the others are
>> extremely helpful during development / debugging / etc. It depends on
>> how "open" you are about HTTP being the transport (i.e. do you let
>> developers augment these headers to support additional features, etc.).
>> This may not make sense in the context of something specialized like
>> Avro transport.
>>> I today implemented a simple HTTP-based transport for Avro: > >
>> https://issues.apache.org/jira...
>> Just out of curiousity, were you using HTTP keep alive? During testing
>> on a project a few years ago, I found a huge difference if Keep Alive is
>> supported. In retrospect, that should have been obvious. I'd imagine the
>> usage pattern here would be a large number of repeated calls between the
>> same client / server within a short period of time; perfect for KA.
>> Regards.
>> --
>> Eric Sammer
>> eric@lifless.net
>> http://esammer.blogspot.com
> --
> Eric Sammer
> eric@lifless.net
> http://esammer.blogspot.com

View raw message