Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Message-ID: <4AC13BC7.6020902@apache.org>
Date: Mon, 28 Sep 2009 15:42:15 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: HTTP transport?
References: <4AAAC403.80809@apache.org>
 <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
In-Reply-To: <1CB467B9-B60C-4ED0-BA82-7BB571E724C6@apache.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Owen O'Malley wrote:
> I've got concerns about this. Both tactical and strategic. The tactical 
> problem is that I need to get security (both Kerberos and token) in to 
> 0.22. I'd really like to get Avro RPC into 0.22. I'd like both to be 
> done roughly in 5 months. If you switch off of the current RPC code base 
> to a completely new RPC code base, I don't see that happening.

What transport do you expect to use with Avro?  If wire-compatibility is 
a goal, and that includes access from languages besides Java, then we 
must use a transport that's well-specified and Java-independent.  HTTP 
is both of these.  The existing Hadoop RPC protocol is not.

We could adapting Hadoop's existing RPC transport to be well-specified 
and language independent.  This is perhaps not a huge task, but it feels 
to me a bit like re-inventing much of what's already in HTTP clients and 
servers these days: connection-pooling, async servers, etc.  Plus we 
take on the onus of fully specifying the transport, so that it may be 
implemented in other languages, and we need to provide some alternate 
implementations to demonstrate this.

Do you feel our existing RPC framework's transport is actually more 
scalable and reliable than, say, Jetty?  Do you think it would be 
substantially harder to add, e.g., token-based security to Jetty than to 
a homegrown server?

> [ HTTP ] also has a couple of disadvantages:
>   - poor integration with kerberos

Do you think it would be substantially harder to integrate Kerberos with 
Jetty than with a homegrown protocol and server?

>   - very expensive on the wire encryption (ssl)

If we don't use HTTP, will we be providing on-wire encryption?  If not, 
this is moot.

Finally, need to have secure HTTP-based access anyway, right?  If we use 
HTTP as our RPC transport mightn't we reuse most of that effort?

Doug