Return-Path: 
 <dev-return-3092-apmail-cassandra-dev-archive=cassandra.apache.org@cassandra.apache.org>
Delivered-To: apmail-cassandra-dev-archive@www.apache.org
Received: (qmail 13862 invoked from network); 30 Dec 2010 10:22:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 30 Dec 2010 10:22:58 -0000
Received: (qmail 35350 invoked by uid 500); 30 Dec 2010 10:22:58 -0000
Delivered-To: apmail-cassandra-dev-archive@cassandra.apache.org
Received: (qmail 35043 invoked by uid 500); 30 Dec 2010 10:22:56 -0000
Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@cassandra.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@cassandra.apache.org>
List-Post: <mailto:dev@cassandra.apache.org>
List-Id: <dev.cassandra.apache.org>
Reply-To: dev@cassandra.apache.org
Delivered-To: mailing list dev@cassandra.apache.org
Received: (qmail 35025 invoked by uid 99); 30 Dec 2010 10:22:55 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Dec 2010 10:22:55 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=10.0
	tests=RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jhodges@twitter.com designates
 209.85.216.44 as permitted sender)
Received: from [209.85.216.44] (HELO mail-qw0-f44.google.com) (209.85.216.44)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Dec 2010 10:22:50 +0000
Received: by qwg5 with SMTP id 5so11455319qwg.31
        for <multiple recipients>; Thu, 30 Dec 2010 02:22:28 -0800 (PST)
MIME-Version: 1.0
Received: by 10.224.2.85 with SMTP id 21mr15087569qai.352.1293704548555; Thu,
 30 Dec 2010 02:22:28 -0800 (PST)
Received: by 10.220.60.140 with HTTP; Thu, 30 Dec 2010 02:22:28 -0800 (PST)
In-Reply-To: <009f01cba808$648e8e60$2dabab20$@com>
References: <1293037253.11611.85.camel@erebus.lan>
	<1293553850.6720.8.camel@erebus.lan>
	<AANLkTinfDjxNmr75Ddiz+reD4pZAe+znuD_hySrsSu-Z@mail.gmail.com>
	<AANLkTimJRXsnGg3XEc51um7nM8_bXcTusHNabnpTM0vb@mail.gmail.com>
	<009f01cba808$648e8e60$2dabab20$@com>
Date: Thu, 30 Dec 2010 02:22:28 -0800
Message-ID: <AANLkTimo23f4FBqZQxowkkjX86877Lh5ZJb-CFeCpTGH@mail.gmail.com>
Subject: Re: Avro RPC?
From: Jeff Hodges <jhodges@twitter.com>
To: dev@cassandra.apache.org
Cc: client-dev@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I think I can help answer that. Avro would be nice because a) it's
going to be part of the core infrastructure of Hadoop and Hadoop is
how most of our long-term stored data is accessed and b) the schema
model is really nice (both keeping the schema in the file with the
serialized data and having a nice definition of how a schema itself is
serialized). Currently, we use lzo-compressed protobuf files to great
effect, instead. See <https://github.com/kevinweil/elephant-bird>.
More importantly, see
<http://www.slideshare.net/kevinweil/hadoop-at-twitter-hadoop-summit-2010?s=
rc=3Drelated_normal&rel=3D4673096>
for why we care about the kind of solutions Avro storage could
provide.

On Thu, Dec 30, 2010 at 2:00 AM, David Dabbs <dmdabbs@gmail.com> wrote:
> Ryan, would you mind pointing us to any doc or history articulating why y=
ou
> feel Avro is preferable for "data storage and anywhere else we
> currently do custom serialization"? Your experiences would be valuable in=
put
> for work I hope to soon begin.
>
> Thank you,
>
> David
>
>
>
>
> -----Original Message-----
> From: Ryan King [mailto:ryan@twitter.com]
> Sent: Tuesday, December 28, 2010 2:10 PM
> To: client-dev@cassandra.apache.org
> Cc: dev@cassandra.apache.org
> Subject: Re: Avro RPC?
>
> On Tue, Dec 28, 2010 at 9:42 AM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
>> On Tue, Dec 28, 2010 at 11:30 AM, Eric Evans <eevans@rackspace.com> wrot=
e:
>>> On Wed, 2010-12-22 at 11:00 -0600, Eric Evans wrote:
>>>> So, Avro RPC. =C2=A0Is anyone using this? =C2=A0Is there anyone intere=
sted in
>>>> seeing it maintained?
>>>>
>>>> I'm concentrating on CQL[1][2], which for me will culminate in the
>>>> creation of a new, application-specific transport, one that doesn't
>>>> use either of the frameworks. =C2=A0To me, the existing RPC framework =
is
>>>> just something to piggy-back on until things are otherwise working,
>>>> and I'm starting to think Thrift might be a better piggy here (read:
>>>> it has more momentum).
>>>
>>> There hasn't been very many people sounding off on this, but those that
>>> have seem to be OK with calling it quits on the Avro interface. =C2=A0S=
ince I
>>> brought this up during the holiday season, I'll give it another week
>>> just in case someone who really cares has been offline.
>>>
>>> To be clear though, I'm not really interested in pushing this forward
>>> anymore, so it's not enough to simply want it, we need someone(s)
>>> willing to step up.
>>>
>>> --
>>> Eric Evans
>>> eevans@rackspace.com
>>>
>>>
>>
>>
>> @Eric I agree with many of your sentiments.
>>
>> The "avro summary" was/is somewhere between wishful thinking and
>> educated guesswork. In ~ May 2010 a shiny new Avro project went top
>> level apache status. Meanwhile thrift had no full time committers and
>> had some glaring bugs that had been open in thrift 0.4.0 (some around
>> php) that annoyed everyone.
>>
>> However thrift did have a release 0.5.0. There are some projects that
>> use thrift, Hbase, Cassandra, and Hive. Thrift still delivers on
>> bindings for a number of languages.
>>
>> Avro is in catchup mode to thrift. They are still evolving the
>> project, while still trying to add support for more languages. As far
>> as I can tell there is no flagship project build around Avro
>> end-to-end.
>>
>> It would be a shame to see the Avro support go away from Cassandra
>> because of all the hard work that was put into it. However the
>> maintenance cost might outweigh it's benefits.
>
> Agreed. Thrift's progress has improved a lot since we first talked
> about using Avro RPC. In the meantime Avro RPC progress has slowed
> (the focus is on the Avro storage implementations).
>
> I think it'd be fair to give up on Avro RPC for client rpc for now*.
> It doesn't deliver enough over Thrift anymore.
>
> -ryan
>
> * I'm still a fan of using avro for data storage and anywhere else we
> currently do custom serialization
>
>
>