Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <AANLkTinwsO+z2kThh1YJN8gpzJBr41LWX=Ar=kcPKDmv@mail.gmail.com>
References: <AANLkTikWx=1VjfPfyJFZXA76=BA=_nK=s5JF5E4iyOKS@mail.gmail.com>
	<AANLkTimG2Dai7-mfqHFqV4=Kqb=-8XTx2_Zg6HSUkOP5@mail.gmail.com>
	<AANLkTinCOKEGpy5DNPee=iOB0AcEARU2cD0Z2Rp=6tHO@mail.gmail.com>
	<AANLkTinwsO+z2kThh1YJN8gpzJBr41LWX=Ar=kcPKDmv@mail.gmail.com>
Date: Wed, 29 Dec 2010 01:03:00 -0800
Message-ID: <AANLkTikzL9FT04teuZNqa-x3DYMzfZucq7nLDWT3ng05@mail.gmail.com>
Subject: Re: Avro Gateway - "Production Quality"
From: Jeff Hammerbacher <hammer@cloudera.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=90e6ba6e867204c1ee049888d6b5

--90e6ba6e867204c1ee049888d6b5
Content-Type: text/plain; charset=ISO-8859-1

Hey Michael,

Wow, very cool. I am a bit embarrassed that I didn't get pyhbase in a better
state before you got to hacking on it, but super happy to have you join the
fun.

I've opened a ticket at https://issues.apache.org/jira/browse/AVRO-721 to
discuss the inclusion of your Tornado client in Avro proper. I think that
would be the right direction for pyhbase. Let's continue the discussion over
there!

Regards,
Jeff

On Mon, Dec 27, 2010 at 5:58 PM, Michael Russo <mjrusso@gmail.com> wrote:

> Jeff, thanks very much for the response and for pushing forward with the
> gateway.
>
> On the Python front, I just wrote an asynchronous client (Tornado-based).
>  Commit:
>
> https://github.com/mjrusso/pyhbase/commit/07ed39527bd6752158b1293e8d2df528d649a613
>
> Let me know if you have any feedback on this.  If you don't feel that it is
> a fit for pyhbase, I'm happy to maintain a fork.  (There's also the
> question
> of if the TornadoRequestor and TornadoHTTPTransceiver should be included in
> the Avro package itself.)
>
> Best,
> Michael
>
> On Sun, Dec 26, 2010 at 3:16 PM, Jeff Hammerbacher <hammer@cloudera.com
> >wrote:
>
> > Okay, I felt guilty about not cleaning this code up in a while.
> >
> > I opened https://issues.apache.org/jira/browse/HBASE-3393 to track
> > updating
> > the server to use the new join() method rather than sleep()ing for a long
> > time. The tests are failing for a very strange reason right now (claiming
> > that a field is missing on an Avro record that's clearly present in the
> > .java file; I'm at a loss), but I'll put my patch up in case you want to
> > try
> > to make progress.
> >
> > Also, Maven can't seem to find the Netty jar needed by Avro. I've got an
> > email out to the Avro lists to figure out what's up with that. For now,
> you
> > can just add the JBoss repository to the HBase pom.xml as directed at
> > http://www.jboss.org/netty/downloads.html.
> >
> > I also fixed a bug where the server wasn't using the port number passed
> at
> > the command line.
> >
> > I also opened https://issues.apache.org/jira/browse/HBASE-3394 to change
> > the
> > build process to generate the Java code at compile time rather than
> forcing
> > us to check in the compiled Java code. That might not be preferred for
> some
> > who prefer source code to be checked in. Happy to discuss on that ticket.
> >
> > Hope that helps,
> > Jeff
> >
> > On Sun, Dec 26, 2010 at 10:50 AM, Jeff Hammerbacher <hammer@cloudera.com
> > >wrote:
> >
> > > Hey Michael,
> > >
> > > Actually, it was something more basic: AVRO-544 changed the way servers
> > are
> > > started so that the gateway class could just join the Avro server
> thread
> > and
> > > run forever, rather than having to run for a finite amount of time. I
> > still
> > > need to get an HBase-side change in that uses this new feature of
> Avro--I
> > > just haven't had much time for coding of late.
> > >
> > > Other than that, I have been lax in keeping up with some of the new
> APIs
> > > introduced in 0.90 (the multi* stuff, for example). There are a few
> other
> > > cleanup tasks at https://issues.apache.org/jira/browse/HBASE-2400 that
> > are
> > > worth checking out.
> > >
> > > Thanks for sending the patches my way for the Python client. I'll do my
> > > best to get to some of these updates soon, and would be more than happy
> > to
> > > review patches for HBase, Avro, or the Python client on Github (thanks
> > for
> > > the patches you've already sent my way).
> > >
> > > Regards,
> > > Jeff
> > >
> > >
> > >
> > > On Sun, Dec 26, 2010 at 7:16 AM, Michael Russo <mjrusso@gmail.com>
> > wrote:
> > >
> > >> Jeff Hammerbacher stated on Quora [1] that "the Avro gateway is not
> > >> production quality" and made a reference to the improvements that an
> > >> upgrade
> > >> to Avro 1.4 would bring.
> > >>
> > >> I would like to understand a little better what this means.
> > >>
> > >> I reviewed the Avro changelog [2] for 1.4+, and very quickly
> > >> scanned o.a.h.h.avro.AvroServer.java [3] and the only Avro change that
> > >> really stood out in the context of significantly improving the
> > production
> > >> quality of the gateway is AVRO-405 (Netty-based Java RPC server) [4].
> > >>
> > >> Is this a reasonable analysis or are there any other important
> elements
> > >> that
> > >> I'm missing?
> > >>
> > >> [1]:
> > >>
> >
> http://www.quora.com/What-is-the-current-status-for-using-Avro-with-HBase
> > >> [2]: https://github.com/apache/avro/blob/trunk/CHANGES.txt
> > >> [3]:
> > >>
> > >>
> >
> http://svn.apache.org/viewvc/hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/AvroServer.java?view=markup
> > >> [4]: https://issues.apache.org/jira/browse/AVRO-405
> > >>
> > >> Thanks,
> > >> Michael
> > >>
> > >
> > >
> >
>

--90e6ba6e867204c1ee049888d6b5--