drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neeraja Rentachintala <nrentachint...@maprtech.com>
Subject Re: Proposal: Create v2 branch to work on breaking changes
Date Tue, 12 Apr 2016 22:38:36 GMT
I agree with Paul. Great points.
I would also add the partners aspect to it. Majority of Drill users use it
in conjunction with a BI tool.


-Neeraja

On Tue, Apr 12, 2016 at 3:34 PM, Paul Rogers <progers@maprtech.com> wrote:

> Hi Jacques,
>
> My two cents…
>
> The unfortunate reality is that enterprise customers move slowly. There is
> a delay in the time it takes for end users to upgrade to a new release.
> When a third-party tool must also upgrade, the delay becomes even longer.
>
> At a high level, we need to provide a window of time in which old/new
> clients work with old/new servers. I may have a 1.6 client. The cluster
> upgrades to 1.8. I need time to upgrade my client to 1.8 — especially if I
> have to wait for the vendor to provide a new package.
>
> If I connect to two clusters, I may upgrade my client to 1.8 for one, but
> I still need to connect to 1.6 for the other if they upgrade on different
> schedules.
>
> This is exactly why we need to figure out a policy: how do we give users a
> sufficient window of time to complete upgrades, even across the 1.x/2.x
> boundary?
>
> The cost of not providing such a window? Broken production systems,
> unpleasant escalations and unhappy customers.
>
> Thanks,
>
> - Paul
>
> > On Apr 12, 2016, at 3:14 PM, Jacques Nadeau <jacques@dremio.com> wrote:
> >
> >>> What I am suggesting is that we need to maintain backward
> compatibility with
> > a defined set of 1.x version clients when Drill 2.0 version is out.
> >
> > I'm asking you to be concrete on why. There is definitely a cost to
> > maintaining this compatibility. What are the real costs if we don't?
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Wed, Apr 6, 2016 at 9:21 AM, Neeraja Rentachintala <
> > nrentachintala@maprtech.com> wrote:
> >
> >> Jacques
> >> can you elaborate on what you mean by 'internal' implementation changes
> but
> >> maintain external API.
> >> I thought that changes that are being discussed here are the Drill
> client
> >> library changes.
> >> What I am suggesting is that we need to maintain backward compatibility
> >> with a defined set of 1.x version clients when Drill 2.0 version is out.
> >>
> >> Neeraja
> >>
> >> On Tue, Apr 5, 2016 at 12:12 PM, Jacques Nadeau <jacques@dremio.com>
> >> wrote:
> >>
> >>> Thanks for bringing this up. BI compatibility is super important.
> >>>
> >>> The discussions here are primarily about internal implementation
> changes
> >> as
> >>> opposed to external API changes. From a BI perspective, I think (hope)
> >>> everyone shares the goal of having zero (to minimal) changes in terms
> of
> >>> ODBC and JDBC behaviors in v2. The items outlined in DRILL-4417 are
> also
> >>> critical to strong BI adoption as numerous patterns right now are
> >>> suboptimal and we need to get them improved.
> >>>
> >>> In terms of your request of the community, it makes sense to have a
> >>> strategy around this. It sounds like you have a bunch of considerations
> >>> that should be weighed but your presentation doesn't actually share
> what
> >>> the concrete details. To date, there has been no formal consensus or
> >>> commitment to any particular compatibility behavior. We've had an
> >> informal
> >>> "don't change wire compatibility within a major version". If we are
> going
> >>> to have a rich dialog about pros and cons of different approaches, we
> >> need
> >>> to make sure that everybody has the same understanding of the dynamics.
> >> For
> >>> example:
> >>>
> >>> Are you saying that someone has packaged the Apache Drill drivers in
> >> their
> >>> BI solution? If so, what version? Is this the Apache release artifact
> or
> >> a
> >>> custom version? Has someone certified them? Did anyone commit a
> >> particular
> >>> compatibility pattern to a BI vendor on behalf of the community?
> >>>
> >>> To date, I'm not aware of any of these types of decisions being
> discussed
> >>> in the community so it is hard to evaluate how important they are
> versus
> >>> other things. Knowing that DRILL-4417 is outstanding and critical to
> the
> >>> best BI experience, I think we should be very cautious of requiring
> >>> long-term support of the existing (internal) implementation.
> Guaranteeing
> >>> ODBC and JDBC behaviors should be satisfactory for the vast majority of
> >>> situations. Anything beyond this needs to have a very public
> cost/benefit
> >>> tradeoff. In other words, please expose your thinking 100x more so that
> >> we
> >>> can all understand the ramifications of different strategies.
> >>>
> >>> thanks!
> >>> Jacques
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Jacques Nadeau
> >>> CTO and Co-Founder, Dremio
> >>>
> >>> On Tue, Apr 5, 2016 at 10:01 AM, Neeraja Rentachintala <
> >>> nrentachintala@maprtech.com> wrote:
> >>>
> >>>> Sorry for coming back to this thread late.
> >>>> I have some feedback on the compatibility aspects of 2.0.
> >>>>
> >>>> We are working with a variety of BI vendors to certify Drill and
> >> provide
> >>>> native connectors for Drill. Having native access from BI tools helps
> >>> with
> >>>> seamless experience for the users with performance and functionality.
> >>> This
> >>>> work is in progress and they are (and will be) working with 1.x
> >> versions
> >>> of
> >>>> Drill as part of the development because thats what we have now. Some
> >> of
> >>>> these connectors will be available before 2.0 and some of them can
> come
> >>> in
> >>>> post 2.0 as certification is a long process. We don't want to be in
a
> >>>> situation where the native connectors are just released by certain BI
> >>>> vendor and the connector is immediately obsolete or doesn't work
> >> because
> >>> we
> >>>> have 2.0 release out now.
> >>>> So the general requirement should be that we maintain backward
> >>>> compatibility with certain number of prior releases. This is very
> >>> important
> >>>> for the success of the project and adoption by eco system. I am happy
> >> to
> >>>> discuss further.
> >>>>
> >>>> -Neeraja
> >>>>
> >>>> On Tue, Apr 5, 2016 at 8:44 AM, Jacques Nadeau <jacques@dremio.com>
> >>> wrote:
> >>>>
> >>>>> I'm going to take this as lazy consensus. I'll create the branch.
> >>>>>
> >>>>> Once created, all merges to the master (1.x branch) should also
go to
> >>> the
> >>>>> v2 branch unless we have a discussion here that they aren't
> >> applicable.
> >>>>> When committing, please make sure to commit to both locations.
> >>>>>
> >>>>> thanks,
> >>>>> Jacques
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Jacques Nadeau
> >>>>> CTO and Co-Founder, Dremio
> >>>>>
> >>>>> On Sat, Mar 26, 2016 at 7:26 PM, Jacques Nadeau <jacques@dremio.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Re Compatibility:
> >>>>>>
> >>>>>> I actually don't even think 1.0 clients work with 1.6 server,
do
> >>> they?
> >>>>>>
> >>>>>> I would probably decrease the cross-compatibility requirement
> >>> burden. A
> >>>>>> nice goal would be cross compatibility across an extended series
of
> >>>>>> releases. However, given all the things we've learned in the
last
> >>> year,
> >>>>> we
> >>>>>> shouldn't try to maintain more legacy than is necessary. As
such, I
> >>>>> propose
> >>>>>> that we consider the requirement of 2.0 to be:
> >>>>>>
> >>>>>> 1.lastX works with 2.firstX. (For example, if 1.8 is the last
minor
> >>>>>> release of the 1.x series, 1.8 would work with 2.0.)
> >>>>>>
> >>>>>> This simplifies testing (we don't have to worry about things
like
> >>> does
> >>>>> 1.1
> >>>>>> work with 2.3, etc) and gives people an upgrade path as they
> >> desire.
> >>>> This
> >>>>>> also allows us to decide what pieces of the compatibility shim
go
> >> in
> >>>> the
> >>>>>> 2.0 server versus the 1.lastX client. (I actually lean towards
> >>>> allowing a
> >>>>>> full break between v1 and v2 server/client but understand that
that
> >>>> level
> >>>>>> or coordination is hard in many organizations since analysts
are
> >>>> separate
> >>>>>> from IT). Hopefully, what I'm proposing can be a good compromise
> >>>> between
> >>>>>> progress and deployment ease.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> Re: Branches/Dangers
> >>>>>>
> >>>>>> Good points on this Julian.
> >>>>>>
> >>>>>> How about this:
> >>>>>>
> >>>>>> - small fixes and enhancements PRs should be made against v1
> >>>>>> - new feature PRs should be made against v2
> >>>>>> - v2 should continue to always pass all precommit tests during
its
> >>> life
> >>>>>> - v2 becomes master in two months
> >>>>>>
> >>>>>> I definitely don't want to create instability in the v2 branch.
> >>>>>>
> >>>>>> The other option I see is we can only do bug fix releases and
> >> branch
> >>>> the
> >>>>>> current master into a maintenance branch and treat master as
v2.
> >>>>>>
> >>>>>> Other ideas?
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Jacques Nadeau
> >>>>>> CTO and Co-Founder, Dremio
> >>>>>>
> >>>>>> On Sat, Mar 26, 2016 at 6:07 PM, Julian Hyde <jhyde@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>>> Do you plan to be doing significant development on both
the v1 and
> >>> v2
> >>>>>>> branches, and if so, for how long? I have been bitten badly
by
> >> that
> >>>>> pattern
> >>>>>>> in the past. Developers put lots of unrelated, destabilizing
> >> changes
> >>>>> into
> >>>>>>> v2, it look longer than expected to stabilize v2, product
> >> management
> >>>>> lost
> >>>>>>> confidence in v2 and shifted resources back to v1, and v2
never
> >>> caught
> >>>>> up
> >>>>>>> with v1.
> >>>>>>>
> >>>>>>> One important question: Which branch will you ask people
to target
> >>> for
> >>>>>>> pull requests? v1, v2 or both? If they submit to v2, and
v2 is
> >>> broken,
> >>>>> how
> >>>>>>> will you know whether the patches are good?
> >>>>>>>
> >>>>>>> My recommendation is to choose one of the following: (1)
put a
> >>> strict
> >>>>>>> time limit of say 2 months after which v2 would become the
master
> >>>> branch
> >>>>>>> (and v1 master would become a maintenance branch), or (2)
make v2
> >>>>> focused
> >>>>>>> on a particular architectural feature; create multiple independent
> >>>>> feature
> >>>>>>> branches with breaking API changes if you need to.
> >>>>>>>
> >>>>>>> Julian
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Mar 26, 2016, at 1:41 PM, Paul Rogers <progers@maprtech.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> 2.0 is a good opportunity to enhance our ZK information.
See
> >>>>>>> DRILL-4543: Advertise Drill-bit ports, status, capabilities
in
> >>>>> ZooKeeper.
> >>>>>>> This change will simplify YARN integration.
> >>>>>>>>
> >>>>>>>> This enhancement will change the “public API” in
ZK. To Parth’s
> >>>> point,
> >>>>>>> we can do so in a way that old clients work - as long as
a
> >> Drill-bit
> >>>>> uses
> >>>>>>> default ports.
> >>>>>>>>
> >>>>>>>> I’ve marked this JIRA as a candidate for 2.0.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> - Paul
> >>>>>>>>
> >>>>>>>>> On Mar 24, 2016, at 4:11 PM, Parth Chandra <parthc@apache.org>
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> What's our proposal for backward compatibility between
1.x and
> >>> 2.x?
> >>>>>>>>> My thoughts:
> >>>>>>>>> Optional  -  Allow a mixture of 1.x and 2.x drillbits
in a
> >>> cluster.
> >>>>>>>>> Required - 1.x clients should be able to talk to
2.x drillbits.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 24, 2016 at 8:55 AM, Jacques Nadeau
<
> >>>> jacques@dremio.com>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> There are some changes that either have reviews
pending or are
> >>> in
> >>>>>>> progress
> >>>>>>>>>> that would require breaking changes to Drill
core.
> >>>>>>>>>>
> >>>>>>>>>> Examples Include:
> >>>>>>>>>> DRILL-4455 (arrow integration)
> >>>>>>>>>> DRILL-4417 (jdbc/odbc/rpc changes)
> >>>>>>>>>> DRILL-4534 (improve null performance)
> >>>>>>>>>>
> >>>>>>>>>> I've created a new 2.0.0 release version in
JIRA and moved
> >> these
> >>>>>>> tasks to
> >>>>>>>>>> that umbrella.
> >>>>>>>>>>
> >>>>>>>>>> I'd like to propose a new v2 release branch
where we can start
> >>>>>>>>>> incorporating these changes without disrupting
v1 stability
> >> and
> >>>>>>>>>> compatibility.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Jacques Nadeau
> >>>>>>>>>> CTO and Co-Founder, Dremio
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message