hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: HTTP transport?
Date Tue, 06 Oct 2009 02:59:24 GMT

On 10/5/09 1:53 PM, "Eric Sammer" <eric@lifeless.net> wrote:

> Ryan:
> 
> Certainly keep alive will help in this case, if that's what you're
> referring to. The server holds the socket for N seconds or M requests,
> which ever comes first. What you're saving with KA is the connection
> setup / tear down. If you have a lot of cases where the client makes a
> single request and goes away, then KA hurts because the server holds the
> connection for the KA timeout (N seconds). This *really* helps if you're
> using TLS due to the additional connection setup overhead.
> 
> It's my opinion and experience that KA helps greatly in the case of many
> exchanges between a small to medium number of clients and a server such
> as RPC. The anti-example is an ad server or web beacon server, for instance.
> 

Even in the beacon case, if the browser is likely to send another request
shortly, it cuts the effective network latency in half.  Establishing a TCP
connection is at minimum one full round trip -- before the request.  If
latency is important KeepAlive is useful as long as a second request is
expected in a short enough time.

As long as the server is not process or thread per connection, one can scale
up connection count rather high (20k) if necessary.

With respect to Avro/Hadoop, I suspect requests from clients to be time
clustered.


> Regards.
> 
> Ryan Rawson wrote:
>> I have a question about these headers... will they impact the ability to do
>> many, but small, rpcs? Imagine you'd need to support 5,000 to 50,000
>> rpcs/second. Would this help or hinder?
>> 
>> On Oct 5, 2009 4:44 PM, "Eric Sammer" <eric@lifeless.net> wrote:
>> 
>> Doug Cutting wrote: > More or less. Except we can probably arrange to omit
>> most of those > response...
>> Content-Type and Server are probably unavoidable. Some of the others are
>> extremely helpful during development / debugging / etc. It depends on
>> how "open" you are about HTTP being the transport (i.e. do you let
>> developers augment these headers to support additional features, etc.).
>> This may not make sense in the context of something specialized like
>> Avro transport.
>> 
>>> I today implemented a simple HTTP-based transport for Avro: > >
>> https://issues.apache.org/jira...
>> Just out of curiousity, were you using HTTP keep alive? During testing
>> on a project a few years ago, I found a huge difference if Keep Alive is
>> supported. In retrospect, that should have been obvious. I'd imagine the
>> usage pattern here would be a large number of repeated calls between the
>> same client / server within a short period of time; perfect for KA.
>> 
>> Regards.
>> --
>> Eric Sammer
>> eric@lifless.net
>> http://esammer.blogspot.com
>> 
> 
> 
> --
> Eric Sammer
> eric@lifless.net
> http://esammer.blogspot.com
> 


Mime
View raw message