drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neeraja Rentachintala <nrentachint...@maprtech.com>
Subject Re: Proposal: Create v2 branch to work on breaking changes
Date Wed, 06 Apr 2016 16:21:26 GMT
Jacques
can you elaborate on what you mean by 'internal' implementation changes but
maintain external API.
I thought that changes that are being discussed here are the Drill client
library changes.
What I am suggesting is that we need to maintain backward compatibility
with a defined set of 1.x version clients when Drill 2.0 version is out.

Neeraja

On Tue, Apr 5, 2016 at 12:12 PM, Jacques Nadeau <jacques@dremio.com> wrote:

> Thanks for bringing this up. BI compatibility is super important.
>
> The discussions here are primarily about internal implementation changes as
> opposed to external API changes. From a BI perspective, I think (hope)
> everyone shares the goal of having zero (to minimal) changes in terms of
> ODBC and JDBC behaviors in v2. The items outlined in DRILL-4417 are also
> critical to strong BI adoption as numerous patterns right now are
> suboptimal and we need to get them improved.
>
> In terms of your request of the community, it makes sense to have a
> strategy around this. It sounds like you have a bunch of considerations
> that should be weighed but your presentation doesn't actually share what
> the concrete details. To date, there has been no formal consensus or
> commitment to any particular compatibility behavior. We've had an informal
> "don't change wire compatibility within a major version". If we are going
> to have a rich dialog about pros and cons of different approaches, we need
> to make sure that everybody has the same understanding of the dynamics. For
> example:
>
> Are you saying that someone has packaged the Apache Drill drivers in their
> BI solution? If so, what version? Is this the Apache release artifact or a
> custom version? Has someone certified them? Did anyone commit a particular
> compatibility pattern to a BI vendor on behalf of the community?
>
> To date, I'm not aware of any of these types of decisions being discussed
> in the community so it is hard to evaluate how important they are versus
> other things. Knowing that DRILL-4417 is outstanding and critical to the
> best BI experience, I think we should be very cautious of requiring
> long-term support of the existing (internal) implementation. Guaranteeing
> ODBC and JDBC behaviors should be satisfactory for the vast majority of
> situations. Anything beyond this needs to have a very public cost/benefit
> tradeoff. In other words, please expose your thinking 100x more so that we
> can all understand the ramifications of different strategies.
>
> thanks!
> Jacques
>
>
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Tue, Apr 5, 2016 at 10:01 AM, Neeraja Rentachintala <
> nrentachintala@maprtech.com> wrote:
>
> > Sorry for coming back to this thread late.
> > I have some feedback on the compatibility aspects of 2.0.
> >
> > We are working with a variety of BI vendors to certify Drill and provide
> > native connectors for Drill. Having native access from BI tools helps
> with
> > seamless experience for the users with performance and functionality.
> This
> > work is in progress and they are (and will be) working with 1.x versions
> of
> > Drill as part of the development because thats what we have now. Some of
> > these connectors will be available before 2.0 and some of them can come
> in
> > post 2.0 as certification is a long process. We don't want to be in a
> > situation where the native connectors are just released by certain BI
> > vendor and the connector is immediately obsolete or doesn't work because
> we
> > have 2.0 release out now.
> > So the general requirement should be that we maintain backward
> > compatibility with certain number of prior releases. This is very
> important
> > for the success of the project and adoption by eco system. I am happy to
> > discuss further.
> >
> > -Neeraja
> >
> > On Tue, Apr 5, 2016 at 8:44 AM, Jacques Nadeau <jacques@dremio.com>
> wrote:
> >
> > > I'm going to take this as lazy consensus. I'll create the branch.
> > >
> > > Once created, all merges to the master (1.x branch) should also go to
> the
> > > v2 branch unless we have a discussion here that they aren't applicable.
> > > When committing, please make sure to commit to both locations.
> > >
> > > thanks,
> > > Jacques
> > >
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Sat, Mar 26, 2016 at 7:26 PM, Jacques Nadeau <jacques@dremio.com>
> > > wrote:
> > >
> > > > Re Compatibility:
> > > >
> > > > I actually don't even think 1.0 clients work with 1.6 server, do
> they?
> > > >
> > > > I would probably decrease the cross-compatibility requirement
> burden. A
> > > > nice goal would be cross compatibility across an extended series of
> > > > releases. However, given all the things we've learned in the last
> year,
> > > we
> > > > shouldn't try to maintain more legacy than is necessary. As such, I
> > > propose
> > > > that we consider the requirement of 2.0 to be:
> > > >
> > > > 1.lastX works with 2.firstX. (For example, if 1.8 is the last minor
> > > > release of the 1.x series, 1.8 would work with 2.0.)
> > > >
> > > > This simplifies testing (we don't have to worry about things like
> does
> > > 1.1
> > > > work with 2.3, etc) and gives people an upgrade path as they desire.
> > This
> > > > also allows us to decide what pieces of the compatibility shim go in
> > the
> > > > 2.0 server versus the 1.lastX client. (I actually lean towards
> > allowing a
> > > > full break between v1 and v2 server/client but understand that that
> > level
> > > > or coordination is hard in many organizations since analysts are
> > separate
> > > > from IT). Hopefully, what I'm proposing can be a good compromise
> > between
> > > > progress and deployment ease.
> > > >
> > > > Thoughts?
> > > >
> > > > Re: Branches/Dangers
> > > >
> > > > Good points on this Julian.
> > > >
> > > > How about this:
> > > >
> > > > - small fixes and enhancements PRs should be made against v1
> > > > - new feature PRs should be made against v2
> > > > - v2 should continue to always pass all precommit tests during its
> life
> > > > - v2 becomes master in two months
> > > >
> > > > I definitely don't want to create instability in the v2 branch.
> > > >
> > > > The other option I see is we can only do bug fix releases and branch
> > the
> > > > current master into a maintenance branch and treat master as v2.
> > > >
> > > > Other ideas?
> > > >
> > > >
> > > > --
> > > > Jacques Nadeau
> > > > CTO and Co-Founder, Dremio
> > > >
> > > > On Sat, Mar 26, 2016 at 6:07 PM, Julian Hyde <jhyde@apache.org>
> wrote:
> > > >
> > > >> Do you plan to be doing significant development on both the v1 and
> v2
> > > >> branches, and if so, for how long? I have been bitten badly by that
> > > pattern
> > > >> in the past. Developers put lots of unrelated, destabilizing changes
> > > into
> > > >> v2, it look longer than expected to stabilize v2, product management
> > > lost
> > > >> confidence in v2 and shifted resources back to v1, and v2 never
> caught
> > > up
> > > >> with v1.
> > > >>
> > > >> One important question: Which branch will you ask people to target
> for
> > > >> pull requests? v1, v2 or both? If they submit to v2, and v2 is
> broken,
> > > how
> > > >> will you know whether the patches are good?
> > > >>
> > > >> My recommendation is to choose one of the following: (1) put a
> strict
> > > >> time limit of say 2 months after which v2 would become the master
> > branch
> > > >> (and v1 master would become a maintenance branch), or (2) make v2
> > > focused
> > > >> on a particular architectural feature; create multiple independent
> > > feature
> > > >> branches with breaking API changes if you need to.
> > > >>
> > > >> Julian
> > > >>
> > > >>
> > > >> > On Mar 26, 2016, at 1:41 PM, Paul Rogers <progers@maprtech.com>
> > > wrote:
> > > >> >
> > > >> > Hi All,
> > > >> >
> > > >> > 2.0 is a good opportunity to enhance our ZK information. See
> > > >> DRILL-4543: Advertise Drill-bit ports, status, capabilities in
> > > ZooKeeper.
> > > >> This change will simplify YARN integration.
> > > >> >
> > > >> > This enhancement will change the “public API” in ZK. To Parth’s
> > point,
> > > >> we can do so in a way that old clients work - as long as a Drill-bit
> > > uses
> > > >> default ports.
> > > >> >
> > > >> > I’ve marked this JIRA as a candidate for 2.0.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > - Paul
> > > >> >
> > > >> >> On Mar 24, 2016, at 4:11 PM, Parth Chandra <parthc@apache.org>
> > > wrote:
> > > >> >>
> > > >> >> What's our proposal for backward compatibility between 1.x
and
> 2.x?
> > > >> >> My thoughts:
> > > >> >> Optional  -  Allow a mixture of 1.x and 2.x drillbits in
a
> cluster.
> > > >> >> Required - 1.x clients should be able to talk to 2.x drillbits.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau <
> > jacques@dremio.com>
> > > >> wrote:
> > > >> >>
> > > >> >>> There are some changes that either have reviews pending
or are
> in
> > > >> progress
> > > >> >>> that would require breaking changes to Drill core.
> > > >> >>>
> > > >> >>> Examples Include:
> > > >> >>> DRILL-4455 (arrow integration)
> > > >> >>> DRILL-4417 (jdbc/odbc/rpc changes)
> > > >> >>> DRILL-4534 (improve null performance)
> > > >> >>>
> > > >> >>> I've created a new 2.0.0 release version in JIRA and
moved these
> > > >> tasks to
> > > >> >>> that umbrella.
> > > >> >>>
> > > >> >>> I'd like to propose a new v2 release branch where we
can start
> > > >> >>> incorporating these changes without disrupting v1 stability
and
> > > >> >>> compatibility.
> > > >> >>>
> > > >> >>>
> > > >> >>> --
> > > >> >>> Jacques Nadeau
> > > >> >>> CTO and Co-Founder, Dremio
> > > >> >>>
> > > >> >
> > > >>
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message