hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashutosh Chauhan <hashut...@apache.org>
Subject Re: Backward incompatible changes
Date Fri, 24 Mar 2017 06:15:53 GMT
So, I just pushed branch-2 from current tip.
Now, lets get 2.2 release out as soon as possible.

Thanks,
Ashutosh

On Thu, Mar 23, 2017 at 9:46 PM, Ashutosh Chauhan <hashutosh@apache.org>
wrote:

> Cutting a branch should not slow down a 2.2 release. If any thing, this
> should help in achieving stabilization faster for release since branch
> won't get any new potentially destabilizing changes but only patches to fix
> existing known issues.
>
> On Thu, Mar 23, 2017 at 8:22 AM, Eugene Koifman <ekoifman@hortonworks.com>
> wrote:
>
>> +1 to make a release first
>>
>> On 3/22/17, 2:06 PM, "Sergey Shelukhin" <sergey@hortonworks.com> wrote:
>>
>>     Hmm.. should we release these first, and then cut branch-2?
>>     Otherwise during the releases, the patches for 2.2/2.3 will need to
>> go to
>>     3 (4?) places (master, branch-2, branch-2.2, branch-2.3?).
>>     There’s no rush to cut the branch if everything in 2.2/2.3 has to go
>> to
>>     3.0 anyway.
>>
>>     On 17/3/22, 13:53, "Pengcheng Xiong" <pxiong@apache.org> wrote:
>>
>>     >I would like to work as the Release Manager if possible. As Owen
>> points
>>     >out, he is working on 2.2 and I will work on 2.3. Thanks.
>>     >
>>     >On Wed, Mar 22, 2017 at 1:32 PM, Ashutosh Chauhan <
>> hashutosh@apache.org>
>>     >wrote:
>>     >
>>     >> Unless there is more feedback, I plan to cut branch-2 in a day or
>> two
>>     >>from
>>     >> current master. As multiple people have suggested on this thread,
>> we
>>     >>should
>>     >> do a 2.2 release soon. Currently there are 177 issues
>>     >> <https://issues.apache.org/jira/issues/?jql=project%20%
>>     >> 3D%20HIVE%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%
>>     >> 5B12310320%5D%20%3D%202.2.0%20ORDER%20BY%20priority%20DESC>
>>     >> targeted for 2.2 release. We can use branch-2 to land these
>> patches and
>>     >>for
>>     >> additional stabilization efforts. Any volunteer for Release Manager
>>     >>driving
>>     >> 2.2 release?
>>     >>
>>     >> Thanks,
>>     >> Ashutosh
>>     >>
>>     >> On Fri, Mar 10, 2017 at 4:23 PM, Ashutosh Chauhan <
>> hashutosh@apache.org>
>>     >> wrote:
>>     >>
>>     >> > I hear what you are saying. Lets begin with 3 concerns:
>>     >> >
>>     >> > - How will we keep the community motivated on fixing both master
>> and
>>     >> > branch-2?
>>     >> > Until we do a stable release from master, stable releases can
>> come
>>     >>only
>>     >> > from branch-2. If a contributor wants to see their fix reach to
>> users
>>     >>on
>>     >> a
>>     >> > stable line quickly they would have to have a fix on branch-2.
>> Also, a
>>     >> > release manager can pick whatever fixes she wants, so even if
>>     >>contributor
>>     >> > doesn't commit it on branch-2, a release manger who wants to do
a
>>     >>release
>>     >> > containing a set of fixes thats always possible.
>>     >> >
>>     >> > - *Harder cherry-picks between master and branch-2*.
>>     >> > That is certainly possible. But hope is we want to keep branch-2
>>     >>stable,
>>     >> > so we don't backport large features which may run into this
>> issue.
>>     >> Smaller
>>     >> > focussed bug fix backport should be possible.
>>     >> >
>>     >> >
>>     >> >    - *Removal of MR2 on the master branch*.
>>     >> > This is something I personally would like to see. But exact
>> timing of
>>     >>it
>>     >> > will be decided by community. I am certainly not saying that as
>> soon
>>     >>as
>>     >> > branch-2 is created, lets remove MR2 on master.
>>     >> >
>>     >> > I would also say that in the end ASF is volunteer organization,
>> we
>>     >>cant
>>     >> > force people to adopt one branch or another. Its upto the
>> contributors
>>     >> what
>>     >> > jiras they work on and when and where they commit it.
>>     >> > By not creating a branch-2 only thing we can guarantee is that
>> rate of
>>     >> > development on master to remain slow because we don't want to
>> start
>>     >>doing
>>     >> > backward incompatible changes without explicitly acknowledging
>> that.
>>     >> >
>>     >> > Thanks,
>>     >> > Ashutosh
>>     >> >
>>     >> > On Thu, Mar 9, 2017 at 12:01 PM, Sergio Pena
>>     >><sergio.pena@cloudera.com>
>>     >> > wrote:
>>     >> >
>>     >> >> Hey Ashutosh, thanks for soliciting feedback on this.
>>     >> >>
>>     >> >> I like the idea you're proposing; maintaining compatibility
and
>> at
>>     >>the
>>     >> >> same time adding newer features to
>>     >> >> Hive consumes a lot of development time and effort.
>>     >> >>
>>     >> >> However, I think some users and companies have just started
to
>> use
>>     >>Hive
>>     >> >> 2.x
>>     >> >> branch as their main major upgrade on Hive
>>     >> >> (possible due to waiting for stabilization and testing
>> upgrades), but
>>     >> >> cutting this major branch that just has 1 year of life
>>     >> >> might make us look like we will forget about the quality of
>> Hive 2.x
>>     >>as
>>     >> we
>>     >> >> did with branch-1.
>>     >> >>
>>     >> >> Hive 1.x latest version was 1.2, and its development stopped
>> because
>>     >>new
>>     >> >> features on Hive 2.x
>>     >> >> Hive 2.x latest version is 2.1, and we want to create Hive
3.x
>>     >>because
>>     >> of
>>     >> >> newer features and incompatibilities.
>>     >> >> Will Hive 3.x have the same future after 3.1 is released?
>>     >> >>
>>     >> >> What I'm also concerned is about these three things:
>>     >> >>
>>     >> >>    - *Branch-2 quality commitment*.
>>     >> >>    How will we keep the community motivated on fixing both
>> master and
>>     >> >>    branch-2?
>>     >> >>    - *Harder cherry-picks between master and branch-2*.
>>     >> >>    Because master will be incompatible by nature, then
>> cherry-picks
>>     >>to
>>     >> >>    branch-2 will be harder.
>>     >> >>    - *Removal of MR2 on the master branch*.
>>     >> >>    This was marked as deprecated just last year, but MR2 is
>> still an
>>     >> >> engine
>>     >> >>    that is used by several users.
>>     >> >>
>>     >> >> I accept that the end of life of major versions will come at
>> some
>>     >>point,
>>     >> >> and these concerns will expire,
>>     >> >> but Hive 2.x is kind of young, isn't it?
>>     >> >>
>>     >> >> Should we try to stabilize the Hive 2.x line first, and have
a
>> few
>>     >>more
>>     >> >> releases before starting to work on Hive 3.0?
>>     >> >> Should we add more test coverage to Hive jenkins jobs to
>> validate
>>     >>Hive
>>     >> 2.x
>>     >> >> quality?
>>     >> >> Should we agree on a date about when we should drop community
>>     >>support on
>>     >> >> Hive versions to let users know about this?
>>     >> >>
>>     >> >> Again, I like your proposal, but I'm afraid that users who
just
>>     >>upgraded
>>     >> >> to
>>     >> >> 2.x won't have any more features and improvements
>>     >> >> because they will be developed on 3.0.
>>     >> >>
>>     >> >> - Sergio
>>     >> >>
>>     >> >>
>>     >> >>
>>     >> >> On Mon, Mar 6, 2017 at 1:24 PM, Ashutosh Chauhan <
>>     >> >> ashutosh.chauhan@gmail.com
>>     >> >> > wrote:
>>     >> >>
>>     >> >> > The way it helps shedding debt  is because dev can now
do
>>     >>refactoring
>>     >> >> > without fear of breaking some rarely used features. The
way
>> that
>>     >>helps
>>     >> >> for
>>     >> >> > adding feature faster is since codebase is lean and easier
to
>>     >>reason
>>     >> >> about
>>     >> >> > its much easier to add new features.
>>     >> >> >
>>     >> >> > More importantly though, it also helps users because we
are
>> setting
>>     >> the
>>     >> >> > expectation from dev community. They can expect that future
>>     >>releases
>>     >> of
>>     >> >> 2.x
>>     >> >> > to be backward compatible. At the same time whenever they
>> decide to
>>     >> >> upgrade
>>     >> >> > they only need to test their application once against
3.x as
>>     >>oppose to
>>     >> >> > continuous breakage of one form or another if we continue
to
>> make
>>     >> >> > incompatible changes in master without branching for 2.x
>>     >> >> >
>>     >> >> > Thanks,
>>     >> >> > Ashutosh
>>     >> >> >
>>     >> >> > On Sat, Mar 4, 2017 at 10:19 AM, Edward Capriolo <
>>     >> edlinuxguru@gmail.com
>>     >> >> >
>>     >> >> > wrote:
>>     >> >> >
>>     >> >> > > Also i dont follow how we remove
>>     >> >> > >
>>     >> >> > > On Saturday, March 4, 2017, Edward Capriolo
>>     >><edlinuxguru@gmail.com>
>>     >> >> > wrote:
>>     >> >> > >
>>     >> >> > > >
>>     >> >> > > >
>>     >> >> > > > On Fri, Mar 3, 2017 at 8:46 PM, Thejas Nair
<
>>     >> thejas.nair@gmail.com
>>     >> >> > > > <javascript:_e(%7B%7D,'cvml','thejas.nair@gmail.com');>>
>> wrote:
>>     >> >> > > >
>>     >> >> > > >> +1
>>     >> >> > > >> There are some features that are incomplete
and what I
>> would
>>     >>not
>>     >> >> > > recommend
>>     >> >> > > >> for any real production use.The 'legacy
authorization
>> mode'
>>     >>is a
>>     >> >> great
>>     >> >> > > >> example of that -
>>     >> >> > > >> https://cwiki.apache.org/confl
>> uence/display/Hive/Hive+Defaul
>>     >> >> > > >> t+Authorization+-+Legacy+Mode
>>     >> >> > > >> . It is inherently insecure mode that nobody
should be
>> using.
>>     >> >> > > >>
>>     >> >> > > >> There is also potential to cleanup of the
thrift api.
>> However,
>>     >> >> there
>>     >> >> > are
>>     >> >> > > >> many users of this api, we would need to
go the
>> deprecation
>>     >>then
>>     >> >> > remove
>>     >> >> > > >> after couple of releases route or so for
that.
>>     >> >> > > >>
>>     >> >> > > >> I am sure there are many other candidates.
We will have
>> to
>>     >> evaluate
>>     >> >> > each
>>     >> >> > > >> of
>>     >> >> > > >> those features on the risk/benefit of keeping
them and
>>     >>arriving
>>     >> at
>>     >> >> a
>>     >> >> > > >> decision.
>>     >> >> > > >>
>>     >> >> > > >> Also, +1 on getting a 2.2 release out before
we branch.
>>     >> >> > > >>
>>     >> >> > > >>
>>     >> >> > > >>
>>     >> >> > > >> On Fri, Mar 3, 2017 at 1:50 PM, Ashutosh
Chauhan <
>>     >> >> > hashutosh@apache.org
>>     >> >> > > >> <javascript:_e(%7B%7D,'cvml','hashutosh@apache.org');>>
>>     >> >> > > >> wrote:
>>     >> >> > > >>
>>     >> >> > > >> > Hi all,
>>     >> >> > > >> >
>>     >> >> > > >> > Hive project has come a long way. With
wide-spread
>> adoption
>>     >> also
>>     >> >> > comes
>>     >> >> > > >> > expectations. Expectation of being
backward compatible
>> and
>>     >>not
>>     >> >> > > breaking
>>     >> >> > > >> > things. However that doesn't come free
of cost and
>> results
>>     >>in
>>     >> >> lot of
>>     >> >> > > >> legacy
>>     >> >> > > >> > code which can't be refactored without
fear of breaking
>>     >>things.
>>     >> >> As a
>>     >> >> > > >> result
>>     >> >> > > >> > project has accumulated lot of debt
over time. At the
>> same
>>     >>time
>>     >> >> > there
>>     >> >> > > >> are
>>     >> >> > > >> > also lot of features which have seen
little uptake. We
>> may
>>     >>want
>>     >> >> to
>>     >> >> > > drop
>>     >> >> > > >> > some of those.
>>     >> >> > > >> >
>>     >> >> > > >> > In order to move forward and shed that
debt we may
>> need a
>>     >>major
>>     >> >> > > version
>>     >> >> > > >> > release which allows us to make backward
incompatible
>>     >>changes
>>     >> and
>>     >> >> > drop
>>     >> >> > > >> > rarely used features. At the same time
there are lots
>> of
>>     >>users
>>     >> >> which
>>     >> >> > > are
>>     >> >> > > >> > consuming currently released 2.1 ,
2.2 branches and
>> expect
>>     >>them
>>     >> >> to
>>     >> >> > > stay
>>     >> >> > > >> on
>>     >> >> > > >> > it for some time. So, I propose that
we create
>> branch-2 from
>>     >> >> current
>>     >> >> > > tip
>>     >> >> > > >> > and do future 2.x releases from that
branch and keep it
>>     >> backward
>>     >> >> > > >> > compatible. This will allow devs to
land breaking
>> changes on
>>     >> >> master
>>     >> >> > > and
>>     >> >> > > >> > pave way to release hive 3.0 in future.
>>     >> >> > > >> >
>>     >> >> > > >> > Ofcourse, each specific incompatible
change and
>> feature drop
>>     >> >> even
>>     >> >> > on
>>     >> >> > > >> > master need to be evaluated on its
own merit on
>>     >>corresponding
>>     >> >> jira.
>>     >> >> > > This
>>     >> >> > > >> > email is just a solicitation of feedback
for creating
>>     >>branch-2
>>     >> >> and
>>     >> >> > > >> allowing
>>     >> >> > > >> > breaking changes in master. Thoughts?
>>     >> >> > > >> >
>>     >> >> > > >> > Thanks,
>>     >> >> > > >> > Ashutosh
>>     >> >> > > >> >
>>     >> >> > > >>
>>     >> >> > > >
>>     >> >> > > > One of the challenges of the developers conducting
the
>>     >> risk-benefit
>>     >> >> > > > analysis are that the developers are mostly
focused on new
>>     >> features,
>>     >> >> > but
>>     >> >> > > > there are deployments of hive that are 5+ years
old and
>> people
>>     >> that
>>     >> >> > rely
>>     >> >> > > on
>>     >> >> > > > the features are not on the mailing list.
>>     >> >> > > >
>>     >> >> > > > For example I developed and use this frequently:
>>     >> >> > > >
>>     >> >> > > > https://community.hortonworks.
>> com/articles/8861/apache-hive-
>>     >> >> > > > groovy-udf-examples.html
>>     >> >> > > >
>>     >> >> > > > My career went away from hive for a while. I
was quite
>>     >>surprised
>>     >> to
>>     >> >> > find
>>     >> >> > > > out the cli->beeline it was more or less
decided not to
>> port
>>     >>it. I
>>     >> >> > > learned
>>     >> >> > > > of this the first time I was forced to work
in a hive
>> server
>>     >>only
>>     >> >> > > > environment and it did not work.
>>     >> >> > > >
>>     >> >> > > > Now I have to go and spend time adding this
back so I
>> don't
>>     >>have
>>     >> to
>>     >> >> > work
>>     >> >> > > > around it not being there.
>>     >> >> > > >
>>     >> >> > > > What we should do continue/doing is making code
that is
>>     >>modular we
>>     >> >> need
>>     >> >> > > to
>>     >> >> > > > break hard dependencies like ThriftSerde or
OrcSerde being
>>     >> "native"
>>     >> >> and
>>     >> >> > > > having to be linked to the metastore move them
out into
>> proper
>>     >> >> > > submodules.
>>     >> >> > > > There is too much code that only works for one
>> implementation
>>     >>of a
>>     >> >> > serde
>>     >> >> > > > etc.
>>     >> >> > > >
>>     >> >> > > >
>>     >> >> > > >
>>     >> >> > >
>>     >> >> > > I would like a timeline to understand this. It sounds
as if
>>     >>master
>>     >> is
>>     >> >> not
>>     >> >> > > releasable currently, so already broken in a way.
We make a
>>     >>branch
>>     >> and
>>     >> >> > > aggreasively break it more?
>>     >> >> > >
>>     >> >> > > Im not following what makes this branching policy
makes
>> adding
>>     >> >> features
>>     >> >> > > faster or how it helps shed debt faster.
>>     >> >> > >
>>     >> >> > >
>>     >> >> > > --
>>     >> >> > > Sorry this was sent from mobile. Will do less grammar
and
>> spell
>>     >> check
>>     >> >> > than
>>     >> >> > > usual.
>>     >> >> > >
>>     >> >> >
>>     >> >>
>>     >> >
>>     >> >
>>     >>
>>
>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message