hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: dynamic RPC and coprocessor changes
Date Wed, 06 Oct 2010 04:03:11 GMT
For what it's worth, the Result changes are just a few functions and
represent a big bang for the buck in terms of lines of code added.  I
fixed the build - sorry about that!

>From what I've seen, the coprocessor stuff interacts but isnt as crazy
as other things.  I'd say go for it.  We only get 1 chance a year to
release apparently.

-ryan

On Tue, Oct 5, 2010 at 8:49 PM, Todd Lipcon <todd@cloudera.com> wrote:
> On Tue, Oct 5, 2010 at 8:17 PM, Andrew Purtell <apurtell@apache.org> wrote:
>
>> > It's not a matter of viewing it as second-class, but more
>> > that it's not necessarily within scope for this exact
>> > release -- in general we have always talked about 0.90 as
>> > a release where we focus on stability and durability.
>>
>> What of replication?
>>
>> Or HFOF?
>>
>>
> Both of these went into our very first 0.89 dev release this summer, and
> have been tried out on clusters for multiple dev releases since. In
> replication's case, it's only been used so far at StumbleUpon AFAIK, but the
> new HFOF support has been used in real workloads at at least 5 companies off
> the top of my head, three of which do not employ any contributors. Like i
> said above, I think new APIs and new big features should go in early in a DR
> series, not right before the release itself.
>
> If we could hold up the 0.90 release for enough time to do a dev release or
> two with coprocessors, I'd be all for it. But given the aims of the release,
> I don't think coprocessors are a blocking feature.
>
>
>> Or the recent Result changes?
>>
>>
> Honestly I've been too swamped to look at these, but if it's a substantial
> new API I also think it should be held for 0.92 as well. If it's a few new
> function calls to existing classes, it seems OK to put it in late.
>
>
>> None are in scope for a stability and durability release.
>>
>> HBASE-2001 has been little changed for 6 months. We've made incremental
>> changes to the server side framework but it's been basically as stable as
>> possible given the changes around it. Then, Gary added a big push with a lot
>> of work on client side support. This part is newer, I grant you that.
>>
>> That it has not received as much review or experience as perhaps you would
>> like means mainly that other HBase committers have had other priorities.
>>
>
> That's true - I've been wanting to review it for months but haven't had time
> to do so. Apologies for that. I think, though, that for this kind of large
> new framework, we want to have some experience actually trying to use it -
> having at least one other person/company outside of TrendMicro use the
> framework to build a non-toy app is the requirement in my mind. I'm pretty
> sure I said exactly that on the very first coprocessor patch, and my opinion
> is the same- it's an awful lot of code that has the primary purpose of
> extensibility, and if we don't have some examples of real extensions, it's
> hard to know if it's all good.
>
> For example, I imagine the Lily people could make good use of this
> framework. Similarly, Sam Pullara's havrobase might provide some good use
> cases. The secondary indexing situation in HBase right now leaves a lot to
> be desired - maybe we could try implementing indexing based on CPs?
>
>
>> That does not mean we do not deserve due consideration.
>>
>>
> Any thoughts on releasing an 0.90-coprocessors-preview next to 0.90, off the
> branch (ie without merging)? I think if we did this, in conjunction with a
> few good blog posts on the hbase blog about example use cases, we could drum
> up some interest and get some feedback, and then merge for 0.92.
>
> -Todd
>
>
>>   - Andy
>>
>> > From: Todd Lipcon <todd@cloudera.com>
>> > Subject: Re: dynamic RPC and coprocessor changes
>> > To: dev@hbase.apache.org
>> > Cc: "apurtell@apache.org" <apurtell@apache.org>
>> > Date: Tuesday, October 5, 2010, 7:57 PM
>> > I agree with Jon here.
>> >
>> > It's not a matter of viewing it as second-class, but more
>> > that it's not
>> > necessarily within scope for this exact release -- in
>> > general we have always
>> > talked about 0.90 as a release where we focus on stability
>> > and durability. A
>> > giant new framework (awesome though it is) seems less
>> > important for this
>> > specific dot release. I also agree that we need to have a
>> > few more use cases
>> > developed to flesh out the API and get more exposure before
>> > merging.
>> >
>> > The idea of the development release series was to give
>> > people exposure to
>> > stuff (and get testing) before it ended up in a so-called
>> > "stable" release.
>> > A large merge should definitely go through at least one or
>> > two dev releases
>> > before showing up in an even-numbered .0. I would also
>> > support a
>> > "coprocessors preview" release done in parallel with 0.90,
>> > available on the
>> > Apache download page, that would let interested people get
>> > involved and try
>> > the branch before it's part of trunk.
>> >
>> > For the same reasons, I also wanted to release
>> > 0.89.20100924 as 0.90 and
>> > push off the new master to 92, but I think I lost that one
>> > :)
>> >
>> > I also agree strongly with Jon that we should immediately
>> > start an 0.91 DR
>> > series with coprocessors merged as soon as 0.90 is
>> > released. We have some
>> > good momentum, and moving quickly towards the next bit will
>> > help keep it up.
>> >
>> > -Todd
>> >
>> > On Tue, Oct 5, 2010 at 7:30 PM, Jonathan Gray <jgray@facebook.com>
>> > wrote:
>> >
>> > > Committing this stuff onto trunk the day before a
>> > feature freeze and as we
>> > > move towards stability gives me a little bit of the
>> > fear.
>> > >
>> > > I completely understand that it's been worked on for
>> > quite some time and
>> > > think this stuff is awesome.  But I still wish I
>> > had more time to give it
>> > > further review and to actually take this stuff for a
>> > test spin.  Right now
>> > > that's just not possible with all the work to be done
>> > stabilizing and
>> > > testing trunk (and I think time is best spent right
>> > now stabilizing the
>> > > master/rs versus new features).
>> > >
>> > > Personally, I have a bunch of fairly sweeping
>> > compaction, flush, and split
>> > > changes that have been partially reviewed but I
>> > haven't had the time to push
>> > > them over the finish line.  I hope to get them in
>> > immediately after an 0.90
>> > > branch is cut and trunk becomes 0.92.  This
>> > should be rather soon.  I'd like
>> > > to see 0.92 get released in short order after 0.90 as
>> > there are lots of
>> > > changes that just have not made it into 0.90
>> > yet.  So even if we waited to
>> > > commit, we could work hard towards the next major
>> > release (I don't think
>> > > there are any major rewrites planned)  :)
>> > >
>> > > With this kind of big change and addition of new APIs,
>> > my preference would
>> > > be to have it committed early in a trunk cycle (versus
>> > immediately before a
>> > > release is cut).  That way, we can play with it
>> > ourselves, see how it works,
>> > > try out some of our own ideas for usage of CPs, etc...
>> > In doing that, I
>> > > would expect there would be suggestions for changes in
>> > the API, additions to
>> > > the API, naming changes to be more clear from a
>> > non-implementer POV, etc...
>> > >
>> > > By putting it in immediately before a release, the
>> > original API can't be
>> > > broken, class names can't be changed, all would need
>> > to be deprecated,
>> > > etc...
>> > >
>> > > The other issue I have with the current CP code that I
>> > have read is the
>> > > number of new classes introduced into existing core
>> > packages (especially
>> > > clent).  Looking at some of their names, they are
>> > generic sounding and it's
>> > > not clear to me without digging in that they are part
>> > of coprocessors.  Is
>> > > it possible to move CP stuff into client.cp packages
>> > and the like?  Or
>> > > prefix with CoProc or some way to keep this stuff
>> > separated?
>> > >
>> > > I apologize for dropping this on you now and if no one
>> > else feels the same,
>> > > I am not going to spoil the party.
>> > >
>> > > But is it critical that this gets into 0.90 and into a
>> > release now?  Could
>> > > we wait for 0.92 and get this committed in a couple
>> > weeks and allowed to
>> > > mature a bit on trunk before it's released?
>> > >
>> > > JG
>> > >
>> > > > -----Original Message-----
>> > > > From: Andrew Purtell [mailto:apurtell@apache.org]
>> > > > Sent: Tuesday, October 05, 2010 7:14 PM
>> > > > To: dev@hbase.apache.org
>> > > > Subject: dynamic RPC and coprocessor changes
>> > > >
>> > > > We've been nursing dynamic RPC and coprocessor
>> > changes for a couple of
>> > > > weeks now and don't see any failures beyond what
>> > are already on trunk
>> > > > with them applied.
>> > > >
>> > > > I was going to apply them tonight but will wait
>> > until tomorrow given
>> > > > that trunk is already disrupted a little.
>> > > >
>> > > > Best regards,
>> > > >
>> > > >     - Andy
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
>>
>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message