hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Devaraj Das <d...@yahoo-inc.com>
Subject Re: HTTP transport?
Date Tue, 29 Sep 2009 23:57:31 GMT
Out of curiosity, do we have such numbers for the current hadoop RPC?


On 9/29/09 4:17 PM, "Doug Cutting" <cutting@apache.org> wrote:

stack wrote:
> So, are we're talking about doing something like following for a
> request/response:
>
>  GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1
>  Host: www.example.com
>
>
>  HTTP/1.1 200 OK
>  Date: Mon, 23 May 2005 22:38:34 GMT
>  Server: Apache/1.3.3.7 (Unix)  (Red-Hat/Linux)
>  Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT
>  Etag: "3f80f-1b6-3e1cb03b"
>  Accept-Ranges: bytes
>  Content-Length: 438
>  Connection: close
>  Content-Type: X-avro/binary
>
>
> ... or some variation on above on each and every RPC?

More or less.  Except we can probably arrange to omit most of those
response headers except Content-Length.  Are any others strictly required?

I today implemented a simple HTTP-based transport for Avro:

   https://issues.apache.org/jira/browse/AVRO-129

In some simple benchmarks I am able to make over 5000 sequential
RPCs/second, each with ~100 bytes of response payload.  Increasing
response payloads to 100kB slows this to around 2500 RPCs/second, giving
throughput of 250MB/second, or 2.5Gbit/s.  This is with both client and
server running on my laptop.  The client is java.net.URLConnection and
the server is Jetty with its default configuration.

Doug


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message