hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Hammerbacher <ham...@cloudera.com>
Subject Re: branching Hive and getting to first release
Date Thu, 26 Mar 2009 18:19:20 GMT
Hey,

What's the state of the release process? We'd really, really like to see a
Hive release. As Nigel said on the Pig list, releasing often is good for the
community.

Thanks,
Jeff

On Tue, Mar 10, 2009 at 2:03 PM, Zheng Shao <zshao9@gmail.com> wrote:

> Till now the discussion is mainly on policy.
>
> However, another important thing is what we plan to work on in the next 2-3
> weeks. Let's say we are focusing on fixing bugs (both Facebook users and
> open-source community users are crying for bug fixes, from what I can see).
> After 2-3 weeks when most bugs are fixed and Hive is more stable, we can
> come back to review this and decide the policy. Given the more information
> we have at that time, we might be able to make a better decision on
> policies.
>
> Also, if we are focusing on fixing bugs for the next 2-3 weeks, there is no
> point to make a 0.3 branch right now because every bug fix will go into
> both
> 0.3 and trunk anyway.
>
> Let's fix most of the important bugs first, then make 0.3 branch, then we
> can work on 2 things at the same time: new features/perf improvement that
> goes only to trunk, and other minor bugs that goes to both 0.3 and trunk.
>
> Thoughts?
>
> Zheng
>
> On Tue, Mar 10, 2009 at 1:09 PM, Ashish Thusoo <athusoo@facebook.com>
> wrote:
>
> > Agreed.
> >
> > I think we moved to trunk because of lazy serde from what Zheng tells me
> (I
> > was out of office when this happened)...
> >
> > Regarding performance fixes, I would rather categorize performance
> > regressions as blocker bugs and keep performance improvements as
> features.
> > By that measure I think lazy serde was fine as a feature. I think we
> should
> > just have let 0.2 stabilize and deployed lazy serde when we released 0.2
> and
> > cut out a 0.3 branch and moved our systems to 0.3. Keeping the criteria
> for
> > what gets categorized as a blocker tight is quite critical otherwise we
> will
> > always be in danger of a constant feature creep and that would totally
> > defeat the purpose of stabilization. In any case if we had been able to
> > stabilize in a months time say for 0.2, I do not think the users would be
> > too unhappy to get the lazy serde a month late. So from that token I
> would
> > not categorize it to be a blocker as such.
> >
> > One constant problem is that the best stress testing environment that we
> > have for Hive right now is our production work load at FB. So I am not
> sure
> > whether we can have a certificate of stability to a branch if we at FB
> pull
> > in patches and run a version that is different from the release. Though
> of
> > course others are always free to get the patches from the JIRA and apply
> > them as they see fit. I am not sure how to address this. Thoughts?
> >
> > Ashish
> >
> > -----Original Message-----
> > From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
> > Sent: Tuesday, March 10, 2009 11:37 AM
> > To: hive-dev@hadoop.apache.org
> > Subject: RE: branching Hive and getting to first release
> >
> > I am in general agreement - but the problems is the mail below doesn't
> > explain why trunk was deployed.
> >
> > Performance fixes are like critical bugs. We cannot run a production
> > cluster that's hurting for performance on non-performant software. To
> that
> > extent - it was a mistake for us to consider lazyserde to be a 'feature'
> > (which is why we didn't back-port it to 0.2). so is hive-223 for example
> -
> > we just need to have it asap in deployment - and by conventional
> definition
> > - it certainly wasn't a regression that would go into a bug fix branch. I
> > suspect there may be more such jiras.
> >
> > One way of looking at this is that we either branched too early, or we
> need
> > to reconsider what goes into a branch.
> >
> > The other way to look at this is that every cluster administrator
> > (including the one at Facebook - who is just like any user of Hive) -
> needs
> > to have the option to pull in latest patches that are critical to his/her
> > deployment. The success of Hive and the happiness of it's internal
> Facebook
> > users should not and cannot be at odds with each other.
> >
> >
> > -----Original Message-----
> > From: Ashish Thusoo [mailto:athusoo@facebook.com]
> > Sent: Tuesday, March 10, 2009 11:08 AM
> > To: hive-dev@hadoop.apache.org
> > Subject: RE: branching Hive and getting to first release
> >
> > I think a big reason for what killed 0.2 was the fact that we decided to
> > deploy trunk into production because of some features that the internal
> > users were asking for, instead of just continuing with the 0.2 branch.
> What
> > I want to stress is that we cannot do that going forward. Once we branch
> out
> > 0.3, we have to let 0.3 soak in production till we have atleast 2 weeks
> of
> > run with no blockers (I did not mean that we will just certify a branch
> to
> > be a relase after 2 weeks - what I meant was that we have at least 2
> weeks
> > of run with no blockers) before we cut out a release from the branch.
> Again
> > I must stress that we have to continue deploying the candidate branch
> into
> > production and we cannot move the production machines to trunk as that
> will
> > completely kill the branch (as happened with 0.2). We have to realy
> isolate
> > blocker bug fixes from features and we have to understand that we cannot
> > role out features overnight (as we have done so far for our users at FB)
> as
> > doing that will make it absolutely hopeless in getting any branch stable.
> >
> > Having said that, we could move to a model where we make a new branch
> (not
> > a release) from trunk once the previous candidate branch is released
> instead
> > of having a train of branches at every 2 weeks. I am fine with that too.
> > What is perhaps more critical is that we have a firm commitment that we
> are
> > not going to deploy new features into production till we stabilize 0.3
> and
> > we should set the expectations accordingly...
> >
> > Ashish
> >
> > -----Original Message-----
> > From: Johan Oskarsson [mailto:johan@oskarsson.nu]
> > Sent: Tuesday, March 10, 2009 9:52 AM
> > To: hive-dev@hadoop.apache.org
> > Subject: Re: branching Hive and getting to first release
> >
> > +1, sounds like a solid plan.
> >
> > Joydeep Sen Sarma wrote:
> > > I am also a little worried about a lot of releases and managing them.
> > perhaps what's clouding my judgement is that there are a lot of critical
> > bugs yet to be fixed - so I don't see how we can stabilize the first
> release
> > in a couple of weeks - or even a month (which is what killed 0.2 I think
> to
> > some extent).
> > >
> > > I would say that the first release is somewhat special. We are fixing a
> > boatload of issues from a very large push of code (all of it!). In
> > subsequent releases - there wouldn't be as many bugs - and a faster
> release
> > cycle would be feasible.
> > >
> > > So my vote would be to branch now (before predicate push down), get the
> > release stable as fast as possible (but potentially wait as long as it
> > takes) - and then only start cutting more branches. Over time - we can
> > converge to a faster release cycle - but right now this seems dubious to
> me.
> > >
> > > Can't put a newborn into kindergarten directly man .. :-)
> > >
> > > -----Original Message-----
> > > From: Johan Oskarsson [mailto:johan@oskarsson.nu]
> > > Sent: Tuesday, March 10, 2009 3:43 AM
> > > To: hive-dev@hadoop.apache.org
> > > Subject: Re: branching Hive and getting to first release
> > >
> > > I'm worried that trying to create a new release every other week will
> > > be too often. Isn't there a risk that we're still fixing bugs in 0.3
> > > when the 0.5 branch is cut if we run into something unexpected?
> > > It seems Hadoop is suffering from this issue a bit lately even though
> > > they branch quarterly, 0.19 still have lots of issues open when people
> > > are committing patches to 0.21 (trunk). Granted Hadoop is a much
> > > larger codebase with more patches applied.
> > >
> > > That said, I won't oppose trying the period suggested and see how it
> > > goes, it's quite easy to change after all.
> > >
> > > /Johan
> > >
> > > Ashish Thusoo wrote:
> > >> For 0.2 we had set a feature freeze date on the 28th of Jan and as I
> > >> had mentioned in the previous email, the plan was cut a branch on the
> > last wednesday of every month and then issue a vote for making it a
> release
> > once it ran satisfactorily (no blocker bugs) for atleast 2 weeks @
> facebook.
> > Accordingly I was hoping that we would limit the changes that would go
> into
> > the branch (0.2) in this case to the blocker bugs only but it seems that
> we
> > had some feature creep and as a result we switched to using trunk at
> > facebook without giving sufficient time for 0.2 to stabilize. It also
> means
> > that perhaps waiting for a month for each release is too long at this
> stage
> > at least for FB. If others are in agreement, how about we do the
> following
> > going forward..
> > >>
> > >>
> > >> Cut a branch every other wednesday, only checkin the most ciritcal
> > blocker bugs into the branch and reserve the features for trunk which
> will
> > be picked up in the next branch and relegiously deploy only the versions
> of
> > the branch at FB. We can start off a vote to make a branch an official
> > release once we have atleast 2 weeks of run on the branch without any
> > blocker bugs (i.e. we did not have a need to upgrade the production
> machines
> > at FB).
> > >>
> > >> We can start off by creating a 0.3 branch this wednesday
> accordingly...
> > >>
> > >> Once we have an agreement on this we can document this procedure on
> the
> > wiki and religiously follow it. Without controlling the tendency of a
> > feature creep it would be difficult to get a stable version out...
> > >>
> > >> Thoughts?
> > >>
> > >> Ashish
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Johan Oskarsson [mailto:johan@oskarsson.nu]
> > >> Sent: Tuesday, March 03, 2009 2:54 AM
> > >> To: hive-dev@hadoop.apache.org
> > >> Subject: Re: branching Hive and getting to first release
> > >>
> > >> To be honest I must've missed that 0.2 was branched (I found the email
> > now though), was there a feature freeze date set?
> > >>
> > >> After branching shouldn't we have moved the non critical issues to 0.3
> > and pushed for fixing the remaining bugs in order to release?
> > >>
> > >> That aside, I don't have a strong opinion whether the next release is
> > >> 0.2 or 0.3, since there hasn't been an Apache release yet. How about
> > setting a feature freeze date now and take it from there?
> > >>
> > >> /Johan
> > >>
> > >> Joydeep Sen Sarma wrote:
> > >>> Hey folks,
> > >>>
> > >>> A few of us were chatting earlier today (some Facebook and Cloudera
> > folks) on best approach to get to a first Hive release.
> > >>>
> > >>> While 0.2 has been branched - it seems awkward to base the first
> > release on it. The reason is twofold:
> > >>>
> > >>> -          new changes to trunk since 0.2 have been relatively
> > contained AFAIK (so no added instability). As evidence - Facebook has
> > reverted to running trunk in production for the last week or so.
> > >>> -          the changes that have gone into trunk since 0.2 are
> > extremely important from performance perspective. This includes the
> > LazySerDe that Zheng added and upcoming hive-232.
> > >>>
> > >>> So one proposal is to branch 0.3 at this point and try to make that
> > first official release for Hive.
> > >>>
> > >>> This does look a little haphazard - and the natural question is
> whether
> > we can stick to this (or we end up repeating this once we throw in some
> more
> > goodies). The feeling is that this may be a good time - hive-279 has
> major
> > changes to the hive compiler and branching 0.3 before those changes are
> > checked in gives us a good chance of producing a stable release with good
> > performance (and the major changes will probably prevent us from
> repeating
> > this trick going forward :)).
> > >>>
> > >>> What do people think?
> > >>>
> > >>> Joydeep
> > >>>
> > >
> >
> >
>
>
> --
> Yours,
> Zheng
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message