avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Gargan <steve.gar...@gmail.com>
Subject Re: Python SocketTransceiver
Date Fri, 19 Nov 2010 08:49:28 GMT

There is a subtle difference between how the ipc works in the http and
socket cases and it boils down to the part the connection plays in
protocol negotiation and the fact that the connection is being reused
in the case of the SocketTranciever and not with the HttpTranciever.

Ideally speaking, once the Requestor and the Responder have negotiated
a protocol between them (the handshake has been completed and a
protocol has been agreed), then so long as the connection is persisted
it is redundant to keep sending the handshake data. In the java
versions of Requestor and Responder the negotiated protocol is stored
along with the connection. The connection is considered 'established'
and so long as it persists, neither side expects the handshake to
appear in any subsequent call between them.  This is all
straightforward enough and works perfectly for the Socket case where
the connection is long lived and maintained across invocations.

In the HTTP case, by the nature of Http, the connection exists only
for a single request and response. There may be http keepalive of the
underlying socket, but conceptually the http connection is closed
after the server responds. Because there is no persistent connection
to associate the negotiated protocol with, a handshake ends up being
sent with every request, though admittedly only hashes are transferred
in the subsequent invocations.

>From what I can tell of the Python requestor, its sending the
handshake every time and while this will work with the HttpServer for
the reason above, it will fail for the SocketServer. In this case, on
the second and subequent invocations, the SocketServer thinks that the
protocol negotiation is complete and is expecting to read the specific
payload data but is instead encountering the retransmitted handshake
header. The error it displays is what you're seeing is what happens
when the server expects to read one thing (like a handshake or
connection metadata) but encounters something else.

In order for your python SocketTranciever to work with the Java
version, it will similarly need to skip sending the handshake once the
connection is established.

Hope this helps.



On Thu, Nov 18, 2010 at 5:10 PM, Eric Evans <eevans@rackspace.com> wrote:
> I put together a SocketTransceiver for Python so that I could compare
> performance against the HTTP one, but I'm having some trouble getting it
> to work.  I'm suspect it's something really dumb, that I'm just too
> close to spot it, and that someone else might.
> The patch against 1.4 is attached.  It causes the Java server to produce
> the following exception:
> java.io.EOFException
>        at
> org.apache.avro.ipc.ByteBufferInputStream.getBuffer(ByteBufferInputStream.java:84)
>        at
> org.apache.avro.ipc.ByteBufferInputStream.read(ByteBufferInputStream.java:46)
>        at org.apache.avro.io.BinaryDecoder
> $InputStreamByteSource.readRaw(BinaryDecoder.java:815)
>        at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:340)
>        at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:265)
>        at
> org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:99)
>        at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:318)
>        at
> org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:229)
>        at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:117)
>        at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
>        at org.apache.avro.ipc.Responder.respond(Responder.java:112)
>        at org.apache.avro.ipc.SocketServer
> $Connection.run(SocketServer.java:91)
>        at java.lang.Thread.run(Thread.java:636)
> Anyone?
> --
> Eric Evans
> eevans@rackspace.com

View raw message