mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gilbert Song <gilb...@mesosphere.io>
Subject Re: Backport Policy
Date Wed, 18 Jul 2018 17:37:55 GMT
Thanks for clarifying the backporting policy, BenM!

I totally agree with the changes proposed for the backporting policy, but I
realize two more scenarios that are more clear to me yet:

   - There are some bugs that are not fixable (due to legacy technical
   decisions), and we end up with fixing the issue by a semantic/behavior
   change in a new release. Do we expect this semantic/behavior change being
   backported?
   - There might be some bugs that root cause is unknown yet, but it did
   impact on a couple releases. If we decide to add some commits for debugging
   purpose (e.g., a new debugging endpoint, or more logging), should we also
   allow these patches to be backported?

For #2, I think we should do the backporting, but for #1, maybe more
discussion is needed since it relates to whether the user has to upgrade or
not.

Cheers,
Gilbert

On Tue, Jul 17, 2018 at 4:26 PM, Lawrence Rau <larryrau@mac.com> wrote:

> I don’t have a big stake in, however, one opinion is if a large commercial
> enterprise was using a specific release that is working the desire is often
> to only upgrade if necessary.  Necessary can be for a number of reasons
> including new features; however if a new feature is not needed the
> compelling reason to upgrade is to fix a specific problem that is causing
> issues.  Thus keeping a maintenance release stable is very important and
> reducing the chance for, while fixing one problem, introducing another.
>
> Often a clear classification of severity of the problem would dictate the
> need to make a change. (yes these can be subjective, but some guidance
> would be better than nothing).
>
> It might be good to give committers guidance on back porting things that
> have a high impact on improving a problem.  Fixing a crashing bug, fixing a
> degenerative performance issue, etc, where these issues have no easy/viable
> work around.  Nice to have fixes aren’t, always, worth updating to.
>
> There can be an argument to respond with a “then don’t upgrade” but if
> changing the release with “nice to have’s” and several point releases later
> when a critical bug is fixed then the org if forced to accept the risk of
> the nice to have’s.
>
> just an opinion.
> …larry
>
>
> On Jul 17, 2018, at 3:00 PM, Chun-Hung Hsiao <chhsiao@mesosphere.io>
> wrote:
>
> I just have a comment on a special case:
> If a fix for a flaky test is easy to backport,
> IMO we probably should backport it,
> otherwise if someone backports another critical fix in the future,
> it would take them extra effort to check all CI failures.
>
> On Mon, Jul 16, 2018 at 11:39 AM Vinod Kone <vinodkone@apache.org> wrote:
>
>> I like how you summarized it Greg and I would vote for leaving the
>> decision
>> to the committer too. In addition to what others mentioned, I think
>> committer should've the responsibility because if things break in a point
>> release (after it is released), it is the committer and contributor who
>> are
>> on the hook to triage and fix it and not the release manager.
>>
>> Having said that, if "during" the release process (i.e., cutting an RC)
>> these backports cause delays for a release manager in getting the release
>> out (e.g., CI flakiness introduced due to backports), release manager
>> could
>> be the ultimate arbiter on whether such a backport should be reverted or
>> fixed by the committer/contributor. Hopefully such issues are caught much
>> before a release process is started (e.g., CI running against release
>> branches).
>>
>> On Mon, Jul 16, 2018 at 1:28 PM Jie Yu <yujie.jay@gmail.com> wrote:
>>
>> > Greg, I like your idea of adding a prescriptive "policy" when evaluating
>> > whether a bug fix should be backported, and leave the decision to
>> committer
>> > (because they have the most context, and avoid a bottleneck in the
>> > process).
>> >
>> > - Jie
>> >
>> > On Mon, Jul 16, 2018 at 11:24 AM, Greg Mann <greg@mesosphere.io> wrote:
>> >
>> > > My impression is that we have two opposing schools of thought here:
>> > >
>> > >    1. Backport as little as possible, to avoid unforeseen consequences
>> > >    2. Backport as much as proves practical, to eliminate bugs in
>> > >    supported versions
>> > >
>> > > Do other people agree with this assessment?
>> > >
>> > > If so, how can we find common ground? One possible solution would be
>> to
>> > > leave the decision on backporting up to the committer, without
>> > specifying a
>> > > project-wide policy. This seems to be the status quo, and would lead
>> to
>> > > some variation across committers regarding what types of fixes are
>> > > backported. We could also choose to delegate the decision to the
>> release
>> > > manager; I favor leaving the decision with the committer, to eliminate
>> > the
>> > > burden on release managers.
>> > >
>> > > Here's a thought: rather than defining a prescriptive "policy" that we
>> > > expect committers to abide by, we could enumerate in the documentation
>> > the
>> > > competing concerns that we expect committers to consider when making
>> > > decisions on backports. The committing docs could read something like:
>> > >
>> > > "When bug fixes are committed to master, the committer should evaluate
>> > the
>> > > fix to determine whether or not it should be backported to supported
>> > > versions. This is left to the committer, but they are expected to
>> weigh
>> > the
>> > > following concerns when making the decision:
>> > >
>> > >    - Every backported change comes with a risk of unintended
>> > >    consequences. The change should be carefully evaluated to ensure
>> that
>> > such
>> > >    side-effects are highly unlikely.
>> > >    - As the complexity of applying a backport increases due to merge
>> > >    conflicts, the likelihood of unintended consequences also
>> increases.
>> > Bug
>> > >    fixes which require extensive rebasing should only be backported
>> when
>> > the
>> > >    bug is critical enough to warrant the risk.
>> > >    - Users of supported versions benefit greatly from the resolution
>> of
>> > >    bugs in point releases. Thus, whenever concerns #1 and #2 can be
>> > allayed
>> > >    for a given bug fix, it should be backported."
>> > >
>> > >
>> > > Cheers,
>> > > Greg
>> > >
>> > >
>> > > On Mon, Jul 16, 2018 at 3:06 AM, Alex Rukletsov <alex@mesosphere.com>
>> > > wrote:
>> > >
>> > >> Back porting as little as possible is the ultimate goal for me. My
>> > >> reasons are closely aligned with what Andrew wrote above.
>> > >>
>> > >> If we agree on this strategy, the next question is how to enforce
>> it. My
>> > >> intuition is that committers will lean towards back porting their
>> > patches
>> > >> in arguable cases, because humans tend to overestimate the
>> importance of
>> > >> their personal work. Delegating the decision in such cases to a
>> release
>> > >> manager in my opinion will help us enforce the strategy of minimal
>> > number
>> > >> backports. As a bonus, the release manager will have a much better
>> > >> understanding of what's going on with the release, keyword: "more
>> > >> ownership".
>> > >>
>> > >> On Sat, Jul 14, 2018 at 12:07 AM, Andrew Schwartzmeyer <
>> > >> andrew@schwartzmeyer.com> wrote:
>> > >>
>> > >>> I believe I fall somewhere between Alex and Ben.
>> > >>>
>> > >>> As for deciding what to backport or not, I lean toward Alex's view
>> of
>> > >>> backporting as little as possible (and agree with his criteria).
My
>> > >>> reasoning is that all changes can have unforeseen consequences,
>> which I
>> > >>> believe is something to be actively avoided in already released
>> > versions.
>> > >>> The reason for backporting patches to fix regressions is the same
as
>> > the
>> > >>> reason to avoid backporting as much as possible: keep behavior
>> > consistent
>> > >>> (and safe) within a release. With that as the goal of a branch
in
>> > >>> maintenance mode, it makes sense to fix regressions, and make
>> > exceptions to
>> > >>> fix CVEs and other critical/blocking issues.
>> > >>>
>> > >>> As for who should decide what to backport, I lean toward Ben's
view
>> of
>> > >>> the burden being on the committer. I don't think we should add
more
>> > work
>> > >>> for release managers, and I think the committer/shepherd obviously
>> has
>> > the
>> > >>> most understanding of the context around changes proposed for
>> backport.
>> > >>>
>> > >>> Here's an example of a recent bugfix which I backported:
>> > >>> https://reviews.apache.org/r/67587/ (for MESOS-3790)
>> > >>>
>> > >>> While normally I believe this change falls under "avoid due to
>> > >>> unforeseen consequences," I made an exception as the bug was old,
>> circa
>> > >>> 2015, (indicating it had been an issue for others), and was causing
>> > >>> recurring failures in testing. The fix itself was very small,
>> meaning
>> > it
>> > >>> was easier to evaluate for possible side effects, so I felt a little
>> > safer
>> > >>> in that regard. The effect of not having the fix was a fatal and
>> > undesired
>> > >>> crash, which furthermore left troublesome side effects on the system
>> > (you
>> > >>> couldn't bring the agent back up). And lastly, a dependent project
>> > (DC/OS)
>> > >>> wanted it in their next bump, which necessitated backporting to
the
>> > release
>> > >>> they were pulling in.
>> > >>>
>> > >>> I think in general we should backport only as necessary, and leave
>> it
>> > on
>> > >>> the committers to decide if backporting a particular change is
>> > necessary.
>> > >>>
>> > >>>
>> > >>> On 07/13/2018 12:54 am, Alex Rukletsov wrote:
>> > >>>
>> > >>>> This is exactly where our views differ, Ben : )
>> > >>>>
>> > >>>> Ideally, I would like a release manager to have more ownership
and
>> > less
>> > >>>> manual work. In my imagination, a release manager has more
power
>> and
>> > >>>> control about dates, features, backports and everything that
is
>> > related
>> > >>>> to
>> > >>>> "their" branch. I would also like us to back port as little
as
>> > >>>> possible, to
>> > >>>> simplify testing and releasing patch versions.
>> > >>>>
>> > >>>> On Fri, Jul 13, 2018 at 1:17 AM, Benjamin Mahler <
>> bmahler@apache.org>
>> > >>>> wrote:
>> > >>>>
>> > >>>> +user, I probably it would be good to hear from users as well.
>> > >>>>>
>> > >>>>> Please see the original proposal as well as Alex's proposal
and
>> let
>> > us
>> > >>>>> know
>> > >>>>> your thoughts.
>> > >>>>>
>> > >>>>> To continue the discussion from where Alex left off:
>> > >>>>>
>> > >>>>> > Other bugs and significant improvements, e.g., performance,
may
>> be
>> > >>>>> back
>> > >>>>> ported,
>> > >>>>> the release manager should ideally be the one who decides
on this.
>> > >>>>>
>> > >>>>> I'm a little puzzled by this, why is the release manager
>> involved? As
>> > >>>>> we
>> > >>>>> already document, backports occur when the bug is fixed,
so this
>> > >>>>> happens in
>> > >>>>> the steady state of development, not at release time. The
release
>> > >>>>> manager
>> > >>>>> only comes in at the time of the release itself, at which
point
>> all
>> > >>>>> backports have already happened and the release manager
handles
>> the
>> > >>>>> release
>> > >>>>> process. Only blocker level issues can stop the release
and while
>> the
>> > >>>>> release manager has a strong say, we should generally agree
on
>> what
>> > >>>>> consists of a release blocking issue.
>> > >>>>>
>> > >>>>> Just to clarify my workflow, I generally backport every
bug fix I
>> > >>>>> commit
>> > >>>>> that applies cleanly, right after I commit it to master
(with the
>> > >>>>> exceptions I listed below).
>> > >>>>>
>> > >>>>> On Thu, Jul 12, 2018 at 8:39 AM, Alex Rukletsov <
>> alex@mesosphere.com
>> > >
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>> > I would like to back port as little as possible. I
suggest the
>> > >>>>> following
>> > >>>>> > criteria:
>> > >>>>> >
>> > >>>>> > * By default, regressions are back ported to existing
release
>> > >>>>> branches. A
>> > >>>>> > bug is considered a regression if the functionality
is present
>> in
>> > the
>> > >>>>> > previous minor or patch version and is not affected
by the bug
>> > there.
>> > >>>>> >
>> > >>>>> > * Critical and blocker issues, e.g., a CVE, can be
back ported.
>> > >>>>> >
>> > >>>>> > * Other bugs and significant improvements, e.g., performance,
>> may
>> > be
>> > >>>>> back
>> > >>>>> > ported, the release manager should ideally be the
one who
>> decides
>> > on
>> > >>>>> this.
>> > >>>>> >
>> > >>>>> > On Thu, Jul 12, 2018 at 12:25 AM, Vinod Kone <
>> vinodkone@apache.org
>> > >
>> > >>>>> wrote:
>> > >>>>> >
>> > >>>>> > > Ben, thanks for the clarification. I'm in agreement
with the
>> > >>>>> points you
>> > >>>>> > > made.
>> > >>>>> > >
>> > >>>>> > > Once we have consensus, would you mind updating
the doc?
>> > >>>>> > >
>> > >>>>> > > On Wed, Jul 11, 2018 at 5:15 PM Benjamin Mahler
<
>> > >>>>> bmahler@apache.org>
>> > >>>>> > > wrote:
>> > >>>>> > >
>> > >>>>> > > > I realized recently that we aren't all on
the same page with
>> > >>>>> > backporting.
>> > >>>>> > > > We currently only document the following:
>> > >>>>> > > >
>> > >>>>> > > > "Typically the fix for an issue that is
affecting supported
>> > >>>>> releases
>> > >>>>> > > lands
>> > >>>>> > > > on the master branch and is then backported
to the release
>> > >>>>> branch(es).
>> > >>>>> > In
>> > >>>>> > > > rare cases, the fix might directly go into
a release branch
>> > >>>>> without
>> > >>>>> > > landing
>> > >>>>> > > > on master (e.g., fix / issue is not applicable
to master)."
>> [1]
>> > >>>>> > > >
>> > >>>>> > > > This leaves room for interpretation about
what lies outside
>> of
>> > >>>>> > "typical".
>> > >>>>> > > > Here's the simplest way I can explain what
I stick to, and
>> I'd
>> > >>>>> like
>> > >>>>> to
>> > >>>>> > > hear
>> > >>>>> > > > what others have in mind:
>> > >>>>> > > >
>> > >>>>> > > > * By default, bug fixes at any level should
be backported to
>> > >>>>> existing
>> > >>>>> > > > release branches if it affects those releases.
Especially
>> > >>>>> important:
>> > >>>>> > > > crashes, bugs in non-experimental features.
>> > >>>>> > > >
>> > >>>>> > > > * Exceptional cases that can omit backporting:
difficult to
>> > >>>>> backport
>> > >>>>> > > fixes
>> > >>>>> > > > (especially if the bugs are deemed of low
priority), bugs in
>> > >>>>> > experimental
>> > >>>>> > > > features.
>> > >>>>> > > >
>> > >>>>> > > > * Exceptional non-bug cases that can be
backported:
>> performance
>> > >>>>> > > > improvements.
>> > >>>>> > > >
>> > >>>>> > > > I realize that there is a ton of subtlety
here (even in
>> terms
>> > of
>> > >>>>> which
>> > >>>>> > > > things are defined as bugs). But I hope
we can lay down a
>> > policy
>> > >>>>> that
>> > >>>>> > > gives
>> > >>>>> > > > everyone the right mindset for common cases
and then discuss
>> > >>>>> corner
>> > >>>>> > cases
>> > >>>>> > > > on-demand in the future.
>> > >>>>> > > >
>> > >>>>> > > > [1] http://mesos.apache.org/documentation/latest/
>> versioning/
>> > >>>>> > > >
>> > >>>>> > >
>> > >>>>> >
>> > >>>>>
>> > >>>>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message