Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BA12B10889 for ; Thu, 6 Jun 2013 18:52:11 +0000 (UTC) Received: (qmail 87462 invoked by uid 500); 6 Jun 2013 18:52:11 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 87404 invoked by uid 500); 6 Jun 2013 18:52:11 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 87391 invoked by uid 99); 6 Jun 2013 18:52:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jun 2013 18:52:10 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of felix.giguere@mate1inc.com designates 74.125.245.92 as permitted sender) Received: from [74.125.245.92] (HELO na3sys010aog112.obsmtp.com) (74.125.245.92) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 06 Jun 2013 18:52:04 +0000 Received: from mail-ve0-f171.google.com ([209.85.128.171]) (using TLSv1) by na3sys010aob112.postini.com ([74.125.244.12]) with SMTP ID DSNKUbDaPqtk8CKiypW/fAGTdWWWzgZbs2/f@postini.com; Thu, 06 Jun 2013 11:51:43 PDT Received: by mail-ve0-f171.google.com with SMTP id b10so2473450vea.2 for ; Thu, 06 Jun 2013 11:51:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=GvvT5hz5MCFEF26gHo4ay9tF3z3fIL5SG7H3ktGd9eQ=; b=J5DGkmjK5J3EEdFBBsquQWVe2y/hm0H/SSNyJwv9WVPDj+tSu7luk4XDRsAJCaqqrf 61Q9j/AqFN7j+F+6iyfR6HMhpNcdXXYEo324e2zOa0h+neTTkB7iY2S3qBHRlEbf+9eD K5vBbD6OZ5aTFXHhOPRAH62AXO8I9UipSrFB9KET+uZnurGRQvv0GUjv6nK6NKyjDhDu a7XK9nfX7GxkNUO+VoEE6QOj4hXq6TPi/Ah0Ztrg/KZFuyPz/54UaZdw5lHU++R02WC3 V7+P0OxzZduT3YkO7IDiTLLkLfpeWoquNWGZRPbrAOAoyKoXe/Kec6i96+hqqkOoIxnx 1W0Q== X-Received: by 10.220.85.204 with SMTP id p12mr22593354vcl.1.1370544701675; Thu, 06 Jun 2013 11:51:41 -0700 (PDT) X-Received: by 10.220.85.204 with SMTP id p12mr22593349vcl.1.1370544701504; Thu, 06 Jun 2013 11:51:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.114.209 with HTTP; Thu, 6 Jun 2013 11:51:16 -0700 (PDT) In-Reply-To: <6A735C3C-0AA3-4F8C-A22D-A8DF25649007@gmail.com> References: <5E1C8602-0CFE-4E42-A4BF-FF9A009AAA7D@gmail.com> <5232CB91-A3A9-48FF-9060-24073622D09D@gmail.com> <6A735C3C-0AA3-4F8C-A22D-A8DF25649007@gmail.com> From: Felix GV Date: Thu, 6 Jun 2013 14:51:16 -0400 Message-ID: Subject: Re: Is Avro right for me? To: user@avro.apache.org Content-Type: multipart/alternative; boundary=001a11c1f53215134404de80cee3 X-Gm-Message-State: ALoCoQkWsnDPj+PKWvV3MmyEigeac64ZBsNgUe5j/Wc+66IdQ3rRXAGebS7VbrfnJP7JzZYUnnkVdDPpbs+Z7mE2mihd2uui/62g9onN4VhyBP0JYQi6QlJETa2TAxwl9PsQ37a2xHVuRRmYdjqiFbDbPrhOhaa+UA== X-Virus-Checked: Checked by ClamAV on apache.org --001a11c1f53215134404de80cee3 Content-Type: text/plain; charset=ISO-8859-1 You can serialize avro messages into json or binary format, and then pass them around to anything else (send them over HTTP, publish them to a message broker system like Kafka or Flume, write them directly into a data store, etc.). You can forget about the avro RPC, as it's just one way amongst many of doing this. You do need to manage schemas properly though. The easy way is to hardcode your schema on both ends, but then that makes it harder to evolve schemas (which avro can do very well otherwise). If you send single serialized avro messages around through a message broker system, then you should definitely consider using a version number for your schema at the beginning of the message, as Martin suggested. Then you can look up what schema each version number represents with something like the versioned schema repo in AVRO-1124 . -- Felix On Tue, Jun 4, 2013 at 11:10 PM, Mark wrote: > I have a question. Say I want to use AVRO as my serialization format to > speak between service applications. Do I need to use AVRO RPC for this or > can I just exchange AVRO messages over HTTP? > > Also whats the difference between an IPC client and an HTTP IPC client? > https://github.com/apache/avro/tree/trunk/lang/ruby/test > > Thanks > > > On May 29, 2013, at 8:02 PM, Mike Percy wrote: > > There is no Ruby support for the Netty Avro RPC protocol that I know of. > But I'm not sure why that matters, other than the fact that the Flume > Thrift support it's not in an official release yet. > > You could also take a look at the Flume HTTP source for a REST-based > interface, but to accept binary data instead of JSON (the default) you > would need to write a small bit of Java code and plug that in. > > Make sure you differentiate between using Avro as a data storage format > and as an RPC mechanism. They are two very different things and don't need > to be tied together. Today, the data storage aspect is more mature and has > much wider language support. > > Mike > > > On Wed, May 29, 2013 at 9:30 AM, Mark wrote: > >> So basically Avro RPC is out of the question? Instead I would need to >> Avro Message -> Thrift -> Flume? Is that along the right lines or am I >> missing something? >> >> >> On May 28, 2013, at 5:02 PM, Mike Percy wrote: >> >> Regarding Ruby support, we recently added support for Thrift RPC, so you >> can now send messages to Flume via Ruby and other non-JVM languages. We >> don't have out-of-the-box client APIs for those yet but would be happy to >> accept patches for it :) >> >> >> > > --001a11c1f53215134404de80cee3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You can serialize avro messages into json or binary format, and then pass t= hem around to anything else (send them over HTTP, publish them to a message= broker system like Kafka or Flume, write them directly into a data store, = etc.). You can forget about the avro RPC, as it's just one way amongst = many of doing this.

You do need to manage schemas properly though. The easy way = is to hardcode your schema on both ends, but then that makes it harder to e= volve schemas (which avro can do very well otherwise). If you send single s= erialized avro messages around through a message broker system, then you sh= ould definitely consider using a version number for your schema at the begi= nning of the message, as Martin suggested. Then you can look up what schema= each version number represents with something like the versioned schema re= po in=A0AVRO-1124= .

--
Felix


On Tue, Jun 4, 2013 at 11:10 PM, Mark <static.void.dev@gmail.com> wrote:
I have a question. =A0Say I want to use= AVRO as my serialization format to speak between service applications. Do = I need to use AVRO RPC for this or can I just exchange AVRO messages over H= TTP?

Also whats the difference between an IPC client and an HTTP = IPC client?=A0https://github.com/apache/avro/tree/trunk/lang/ru= by/test

Thanks


On = May 29, 2013, at 8:02 PM, Mike Percy <mpercy@apache.org> wrote:

There is no Ruby support for the Netty Avro RPC protocol t= hat I know of. But I'm not sure why that matters, other than the fact t= hat the Flume Thrift support it's not in an official release yet.

You could also take a look at the Flume HTTP source for a RE= ST-based interface, but to accept binary data instead of JSON (the default)= you would need to write a small bit of Java code and plug that in.

Make sure you differentiate between using Avro as a dat= a storage format and as an RPC mechanism. They are two very different thing= s and don't need to be tied together. Today, the data storage aspect is= more mature and has much wider language support.

Mike


On Wed, May 29, 2013 at 9:30 AM, Mark &l= t;static.voi= d.dev@gmail.com> wrote:
So basic= ally Avro RPC is out of the question? Instead I would need to Avro Message = -> Thrift -> Flume? Is that along the right lines or am I missing som= ething?


On May 28, 2013, at 5:02 PM, Mike Percy <mpercy@apache.org> w= rote:

Regarding Ruby support, we recently added support= for Thrift RPC, so you can now send messages to Flume via Ruby and other n= on-JVM languages. We don't have out-of-the-box client APIs for those ye= t but would be happy to accept patches for it :)



--001a11c1f53215134404de80cee3--