From general-return-539-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Mon Sep 28 20:14:40 2009 Return-Path: Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: (qmail 95734 invoked from network); 28 Sep 2009 20:14:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 28 Sep 2009 20:14:40 -0000 Received: (qmail 12104 invoked by uid 500); 28 Sep 2009 20:14:39 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 12026 invoked by uid 500); 28 Sep 2009 20:14:39 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 12016 invoked by uid 99); 28 Sep 2009 20:14:39 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 20:14:39 +0000 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Sep 2009 20:14:27 +0000 Received: from oceanfarearth-lm.corp.yahoo.com (oceanfarearth-lm.corp.yahoo.com [10.72.113.156]) by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n8SKD42o001720 for ; Mon, 28 Sep 2009 13:13:04 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=message-id:from:to:in-reply-to:content-type:mime-version: subject:date:references:x-mailer; b=yFWMrq/7IUlyqH+eKsPNrVGcYXZTWKD+Hf4YqY7QydmSuyr9UKICLBGE4Wo50Ilr Message-Id: From: Sanjay Radia To: In-Reply-To: <4AAAC403.80809@apache.org> Content-Type: multipart/alternative; boundary=Apple-Mail-36-983677215 Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: HTTP transport? Date: Mon, 28 Sep 2009 13:13:04 -0700 References: <4AAAC403.80809@apache.org> X-Mailer: Apple Mail (2.936) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-36-983677215 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Sep 11, 2009, at 2:41 PM, Doug Cutting wrote: > I'm considering an HTTP-based transport for Avro as the preferred, > high-performance option. > > HTTP has lots of advantages. In particular, it already has > - lots of authentication, authorization and encryption support; > - highly optimized servers; > - monitoring, logging, etc. > Q. Is this to replace the client-DN data-transfer protocol or for ALL Hadoop rpc? Q. Was authentication one of your main motivation? The current plans for authentication is centered around kerberos. HTTP does not fit in too well in that picture. sanjay > > Tomcat and other servlet containers support async NIO, where a > thread is > not required per connection. A servlet can process bulk data with a > single copy to and from the socket (bypassing stream buffers). Calls > can be multiplexed over a single HTTP connection using Comet events. > > http://tomcat.apache.org/tomcat-6.0-doc/aio.html > > Zero copy is not an option for servlets that generate arbitrary data, > but one can specify a file/start/length tuple and Tomcat will use > sendfile to write the response. That means that while HDFS datanode > file reads could not be done via RPC, they could be done via HTTP with > zero-copy. If authentication and authorization are already done in > the > HTTP server, this may not be a big loss. The HDFS client might make > two > HTTP requests, one to read a files data, and another to read its > checksums. The server would then stream the entire block to the > client > using sendfile, using TCP flow control as today. > > Thoughts? > > Doug > --Apple-Mail-36-983677215--