drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: Proposal: Create v2 branch to work on breaking changes
Date Tue, 05 Apr 2016 19:12:24 GMT
Thanks for bringing this up. BI compatibility is super important.

The discussions here are primarily about internal implementation changes as
opposed to external API changes. From a BI perspective, I think (hope)
everyone shares the goal of having zero (to minimal) changes in terms of
ODBC and JDBC behaviors in v2. The items outlined in DRILL-4417 are also
critical to strong BI adoption as numerous patterns right now are
suboptimal and we need to get them improved.

In terms of your request of the community, it makes sense to have a
strategy around this. It sounds like you have a bunch of considerations
that should be weighed but your presentation doesn't actually share what
the concrete details. To date, there has been no formal consensus or
commitment to any particular compatibility behavior. We've had an informal
"don't change wire compatibility within a major version". If we are going
to have a rich dialog about pros and cons of different approaches, we need
to make sure that everybody has the same understanding of the dynamics. For
example:

Are you saying that someone has packaged the Apache Drill drivers in their
BI solution? If so, what version? Is this the Apache release artifact or a
custom version? Has someone certified them? Did anyone commit a particular
compatibility pattern to a BI vendor on behalf of the community?

To date, I'm not aware of any of these types of decisions being discussed
in the community so it is hard to evaluate how important they are versus
other things. Knowing that DRILL-4417 is outstanding and critical to the
best BI experience, I think we should be very cautious of requiring
long-term support of the existing (internal) implementation. Guaranteeing
ODBC and JDBC behaviors should be satisfactory for the vast majority of
situations. Anything beyond this needs to have a very public cost/benefit
tradeoff. In other words, please expose your thinking 100x more so that we
can all understand the ramifications of different strategies.

thanks!
Jacques





--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Apr 5, 2016 at 10:01 AM, Neeraja Rentachintala <
nrentachintala@maprtech.com> wrote:

> Sorry for coming back to this thread late.
> I have some feedback on the compatibility aspects of 2.0.
>
> We are working with a variety of BI vendors to certify Drill and provide
> native connectors for Drill. Having native access from BI tools helps with
> seamless experience for the users with performance and functionality. This
> work is in progress and they are (and will be) working with 1.x versions of
> Drill as part of the development because thats what we have now. Some of
> these connectors will be available before 2.0 and some of them can come in
> post 2.0 as certification is a long process. We don't want to be in a
> situation where the native connectors are just released by certain BI
> vendor and the connector is immediately obsolete or doesn't work because we
> have 2.0 release out now.
> So the general requirement should be that we maintain backward
> compatibility with certain number of prior releases. This is very important
> for the success of the project and adoption by eco system. I am happy to
> discuss further.
>
> -Neeraja
>
> On Tue, Apr 5, 2016 at 8:44 AM, Jacques Nadeau <jacques@dremio.com> wrote:
>
> > I'm going to take this as lazy consensus. I'll create the branch.
> >
> > Once created, all merges to the master (1.x branch) should also go to the
> > v2 branch unless we have a discussion here that they aren't applicable.
> > When committing, please make sure to commit to both locations.
> >
> > thanks,
> > Jacques
> >
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Sat, Mar 26, 2016 at 7:26 PM, Jacques Nadeau <jacques@dremio.com>
> > wrote:
> >
> > > Re Compatibility:
> > >
> > > I actually don't even think 1.0 clients work with 1.6 server, do they?
> > >
> > > I would probably decrease the cross-compatibility requirement burden. A
> > > nice goal would be cross compatibility across an extended series of
> > > releases. However, given all the things we've learned in the last year,
> > we
> > > shouldn't try to maintain more legacy than is necessary. As such, I
> > propose
> > > that we consider the requirement of 2.0 to be:
> > >
> > > 1.lastX works with 2.firstX. (For example, if 1.8 is the last minor
> > > release of the 1.x series, 1.8 would work with 2.0.)
> > >
> > > This simplifies testing (we don't have to worry about things like does
> > 1.1
> > > work with 2.3, etc) and gives people an upgrade path as they desire.
> This
> > > also allows us to decide what pieces of the compatibility shim go in
> the
> > > 2.0 server versus the 1.lastX client. (I actually lean towards
> allowing a
> > > full break between v1 and v2 server/client but understand that that
> level
> > > or coordination is hard in many organizations since analysts are
> separate
> > > from IT). Hopefully, what I'm proposing can be a good compromise
> between
> > > progress and deployment ease.
> > >
> > > Thoughts?
> > >
> > > Re: Branches/Dangers
> > >
> > > Good points on this Julian.
> > >
> > > How about this:
> > >
> > > - small fixes and enhancements PRs should be made against v1
> > > - new feature PRs should be made against v2
> > > - v2 should continue to always pass all precommit tests during its life
> > > - v2 becomes master in two months
> > >
> > > I definitely don't want to create instability in the v2 branch.
> > >
> > > The other option I see is we can only do bug fix releases and branch
> the
> > > current master into a maintenance branch and treat master as v2.
> > >
> > > Other ideas?
> > >
> > >
> > > --
> > > Jacques Nadeau
> > > CTO and Co-Founder, Dremio
> > >
> > > On Sat, Mar 26, 2016 at 6:07 PM, Julian Hyde <jhyde@apache.org> wrote:
> > >
> > >> Do you plan to be doing significant development on both the v1 and v2
> > >> branches, and if so, for how long? I have been bitten badly by that
> > pattern
> > >> in the past. Developers put lots of unrelated, destabilizing changes
> > into
> > >> v2, it look longer than expected to stabilize v2, product management
> > lost
> > >> confidence in v2 and shifted resources back to v1, and v2 never caught
> > up
> > >> with v1.
> > >>
> > >> One important question: Which branch will you ask people to target for
> > >> pull requests? v1, v2 or both? If they submit to v2, and v2 is broken,
> > how
> > >> will you know whether the patches are good?
> > >>
> > >> My recommendation is to choose one of the following: (1) put a strict
> > >> time limit of say 2 months after which v2 would become the master
> branch
> > >> (and v1 master would become a maintenance branch), or (2) make v2
> > focused
> > >> on a particular architectural feature; create multiple independent
> > feature
> > >> branches with breaking API changes if you need to.
> > >>
> > >> Julian
> > >>
> > >>
> > >> > On Mar 26, 2016, at 1:41 PM, Paul Rogers <progers@maprtech.com>
> > wrote:
> > >> >
> > >> > Hi All,
> > >> >
> > >> > 2.0 is a good opportunity to enhance our ZK information. See
> > >> DRILL-4543: Advertise Drill-bit ports, status, capabilities in
> > ZooKeeper.
> > >> This change will simplify YARN integration.
> > >> >
> > >> > This enhancement will change the “public API” in ZK. To Parth’s
> point,
> > >> we can do so in a way that old clients work - as long as a Drill-bit
> > uses
> > >> default ports.
> > >> >
> > >> > I’ve marked this JIRA as a candidate for 2.0.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > - Paul
> > >> >
> > >> >> On Mar 24, 2016, at 4:11 PM, Parth Chandra <parthc@apache.org>
> > wrote:
> > >> >>
> > >> >> What's our proposal for backward compatibility between 1.x and
2.x?
> > >> >> My thoughts:
> > >> >> Optional  -  Allow a mixture of 1.x and 2.x drillbits in a cluster.
> > >> >> Required - 1.x clients should be able to talk to 2.x drillbits.
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau <
> jacques@dremio.com>
> > >> wrote:
> > >> >>
> > >> >>> There are some changes that either have reviews pending or
are in
> > >> progress
> > >> >>> that would require breaking changes to Drill core.
> > >> >>>
> > >> >>> Examples Include:
> > >> >>> DRILL-4455 (arrow integration)
> > >> >>> DRILL-4417 (jdbc/odbc/rpc changes)
> > >> >>> DRILL-4534 (improve null performance)
> > >> >>>
> > >> >>> I've created a new 2.0.0 release version in JIRA and moved
these
> > >> tasks to
> > >> >>> that umbrella.
> > >> >>>
> > >> >>> I'd like to propose a new v2 release branch where we can start
> > >> >>> incorporating these changes without disrupting v1 stability
and
> > >> >>> compatibility.
> > >> >>>
> > >> >>>
> > >> >>> --
> > >> >>> Jacques Nadeau
> > >> >>> CTO and Co-Founder, Dremio
> > >> >>>
> > >> >
> > >>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message