hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sammer <e...@lifeless.net>
Subject Re: HTTP transport?
Date Mon, 05 Oct 2009 21:13:51 GMT
Ryan Rawson wrote:
> That's good to know. I thought ka would help... but I was also talking about
> the overhead of a header where the payload is smaller than the framing. Eg:
> 8 byte requests, excluding which rpc. This seems like we could be hurt since
> the headers are potentially 5x the size of our payload/request params.

Oh, got you. That's the classic SOAP problem. ;) I think it's possible,
but to what degree I couldn't be sure. HTTP has that kind of overhead
because of the generality. I think you'd get that with anything that
isn't specifically designed to be wire efficient. Of course, you wind up
having to do what Doug originally mentioned; rebuilding and maintaining
the original stuff that HTTP (and the supporting clients) already supports.

If over-optimization is in fact the root of all evil (which I've heard
once or twice) then maybe it makes sense to start simple and iterate if
necessary. In other words, say screw it, use HTTP, strip unnecessary
headers, but design the code such that Avro's transport is interface
based in case it needs to change. I think prototyping an Avro transport
with HTTP, optimizing, and then dropping lower level if necessary is a
better approach than going straight to the latter.

All of that said, I don't have the insight into the code base that some
of the other folks do. This is based on my experience with similar high
throughput systems, but I wouldn't say I'm 100% convinced it applies
here as the payloads in those systems were bigger than 8 bytes.

Eric Sammer

View raw message