avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Krawczyk <ste...@nextdoor.com>
Subject Re: Python/Java/Ruby RPC interop issues
Date Tue, 25 Jun 2013 00:30:20 GMT
Hi Steve,

I'm new to the Avro community as well, but it just seems that nobody really
cares about Avro support outside of Java/C++...

The issue you're seeing is because the python & ruby implementations are
using the HTTPTransceiver, whereas the Java server is using the
NettyTransceiver. They dont' talk the same language. I'm as surprised as
you are that such a change would be introduced without the other languages
having support (or providing documentation mentioning that they don't work).

I asked this list earlier what it would take to get python to talk netty
but nobody responded...

We're also using Flume, and the only reasonable option is to use Thrift RPC
& the Thrift Source (look at 1.4 branch of code) as the ingestion point
from our app. From there the rest of the flume pipeline can use Avro RPC.
The other option is to use the legacy avro source, but that source is
effectively deprecated...

Hope that helps.



On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams

> Hello!
> New to the Avro world. I have been evaluating Flume and testing various
> capabilities. I have run into an issue with the Avro RPC piece. So far, I
> have found that the Flume Avro RPC java client works with a Flume Avro
> Source as expected (I believe these implementations use Netty). However,
> when I tried using a Ruby RPC client to communicate with the Flume Avro
> Source, there was a failure and the source reported:
> org.apache.avro.AvroRuntimeException: Excessively large list allocation
> request detected: 539959368 items! Connection closed.
> I then tried this with a Python RPC client and had a similar issue. I
> suspected client/server Avro interop issues, so then verified that a Python
> Avro RPC client and Python Avro RPC server communicate correctly (using
> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby
> Avro RPC client and Ruby Avro RPC server communicate correctly. However,
> using the Python Avro RPC client with Ruby Avro RPC server fails (client
> hangs), and Ruby client with Python server also fails (again, client hangs).
> I know I'm not providing exhaustive info here, but I'm surprised to find
> out-of-the-box interop problems. Do folks regularly use these
> client/servers in various mixes (Java/Ruby/Python)? Are there particular
> gotchas or configurations that are not obvious?
> Thank you,
> Steve

View raw message