Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 11756 invoked from network); 29 Sep 2009 23:58:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 29 Sep 2009 23:58:53 -0000 Received: (qmail 86239 invoked by uid 500); 29 Sep 2009 23:58:52 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 86137 invoked by uid 500); 29 Sep 2009 23:58:52 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 86127 invoked by uid 99); 29 Sep 2009 23:58:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:58:52 +0000 X-ASF-Spam-Status: No, hits=4.2 required=10.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [69.147.107.21] (HELO mrout2-b.corp.re1.yahoo.com) (69.147.107.21) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 29 Sep 2009 23:58:41 +0000 Received: from sp1-ex07cas03.ds.corp.yahoo.com (sp1-ex07cas03.ds.corp.yahoo.com [216.252.116.151]) by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n8TNvYcx021942 for ; Tue, 29 Sep 2009 16:57:34 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:from:to:date:subject:thread-topic:thread-index: message-id:in-reply-to:accept-language:content-language: x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version; b=xnK6zAWp8In3PbZakCVUYO5X2dG3Js/PvF0mjJLdx0TTrXxSvg1knw+uh9BI8p5X Received: from SP1-EX07VS02.ds.corp.yahoo.com ([216.252.116.135]) by sp1-ex07cas03.ds.corp.yahoo.com ([216.252.116.151]) with mapi; Tue, 29 Sep 2009 16:57:33 -0700 From: Devaraj Das To: "general@hadoop.apache.org" Date: Tue, 29 Sep 2009 16:57:31 -0700 Subject: Re: HTTP transport? Thread-Topic: HTTP transport? Thread-Index: AcpBWywlZmAutc7dTNSnntYSTy7nawABW8OE Message-ID: In-Reply-To: <4AC2956F.6020303@apache.org> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C6E7ECFBD5F0ddasyahooinccom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C6E7ECFBD5F0ddasyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Out of curiosity, do we have such numbers for the current hadoop RPC? On 9/29/09 4:17 PM, "Doug Cutting" wrote: stack wrote: > So, are we're talking about doing something like following for a > request/response: > > GET /avro/org.apache.hadoop.hbase.RegionServer HTTP/1.1 > Host: www.example.com > > > HTTP/1.1 200 OK > Date: Mon, 23 May 2005 22:38:34 GMT > Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux) > Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT > Etag: "3f80f-1b6-3e1cb03b" > Accept-Ranges: bytes > Content-Length: 438 > Connection: close > Content-Type: X-avro/binary > > > ... or some variation on above on each and every RPC? More or less. Except we can probably arrange to omit most of those response headers except Content-Length. Are any others strictly required? I today implemented a simple HTTP-based transport for Avro: https://issues.apache.org/jira/browse/AVRO-129 In some simple benchmarks I am able to make over 5000 sequential RPCs/second, each with ~100 bytes of response payload. Increasing response payloads to 100kB slows this to around 2500 RPCs/second, giving throughput of 250MB/second, or 2.5Gbit/s. This is with both client and server running on my laptop. The client is java.net.URLConnection and the server is Jetty with its default configuration. Doug --_000_C6E7ECFBD5F0ddasyahooinccom_--