cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Burwell <>
Subject Re: [PROPOSAL] LTS Release Cycle
Date Tue, 02 Feb 2016 16:40:42 GMT

Based on the feedback from Ilya, Erik, and Daan, I have updated my original LTS proposal to
clarify that LTS releases are official project deliverables, commit traceability across branches,
and RM approval of PRs:

## START ##


The current monthly release cycle addresses the needs of users focused on deploying new functionality
as quickly as possible.  It does not address the needs of users oriented towards stability
rather than new functionality.  These users typically employ QA processes to comply with corporate
policy and/or regulatory requirements.  To maintain a growing, thriving community, we must
address the needs of both user types.  Therefore, I propose that we overlay a LTS release
cycle onto the monthly release cycle to address the needs of stability-oriented users with
minimal to no impact on the monthly release cycle.  This proposed LTS release cycle has the
following goals:

  * Prefer Stability to New Functionality: Deliver releases that only address defects and
CVEs.  This narrowly focused change scope greatly reduces the upgrade risk/operational impact
and shorter internal QA cycles.
  * Reliable Release Lifetimes: Embracing a time-based release strategy, the LTS release cycle
will provide users with a reliable support time frames.  Users can use these time frames provide
users with an 20 month window in which to plan upgrades.
  * Support Sustainability: With a defined end of support for LTS releases and a maximum of
two (2) LTS releases under active maintenance at any given time, community members can better
plan their commitments to release support activities.  We also have a agreed upon policy for
release end-of-life (EOL) to debate about continuing work on old releases.

LTS releases would be official project releases.  Therefore, they would be subject to same
release voting requirements and available from the project downloads page.

Proposed Process

LTS release branches will be cut twice year on 1 Jan and 1 July based the tag of the most
recent monthly release.  The branch will be named <base version>_LTS and each LTS release
will be versioned in the form of <base version>_<LTS revision number>.  For example,
if we cut an LTS branch based on 4.7.0, the branch would be named 4.7.0_LTS and the version
of the first LTS release would be 4.7.0_0, the second would be 4.7.0_1, etc.  This release
naming convention differentiates LTS and monthly releases, communicates the version on which
the LTS release is based, and allows the maintenance releases for monthly releases without
version number contention/conflict.  Finally, like master, an LTS branch would be always deployable
following its initial release.  While it is unlikely that LTS users would deploy from the
branch, the quality discipline of this requirement will benefit the long term stability of
LTS releases.  Like master, all PRs targeting an LTS branch would require two LGTMs (one code
review and one independent test), as well as, an LGTM from the branch RM.  A combined code
review/test LGTM and an RM LGTM would be acceptable.

The following are the types of changes that would permitted and guarantees provided to users:

  * No features or enhancements would be backported to LTS release branches.
  * Database changes would be limited to those required to address the backported defect fixes.
  * Support for the release/version of the following components from the release on which
the LTS is based throughout the entire release cycle:
    * MySQL/MariaDB
    * JDK/JRE
    * Linux distributions
  * API compatibility for between all LTS revisions.  API changes would be limited to those
required to fix defects or address security issues.

An LTS release would have a twenty (20) month lifetime from the date the release branch is
cut.  This support period allows up to two (2) months of branch stabilization before initial
release with a minimum of eighteen (18) months of availability for deployment.  The twenty
(20) month LTS lifecycle would be divided into following support periods:

  * 0-2 months (average): Stablization of the LTS branch with fixes based on defects discovered
from functional, regression, endurance, and scalability testing.
  * 2-14 months: backport blocker and critical priority defect fixes in the scope of the LTS
branch functionality; fix all blocker and critical priority defects identified on the LTS
  * 14-20 months: backport only CVE fixes in the scope of the LTS branch functionality; fix
all blocker priority defects identified on the LTS branch

Defect fixes that originate in an LTS branch would be pulled forward to master only.  Finally,
an LTS branch should be release the fewest times necessary to deliver fixes in a relatively
timely manner.  Therefore, LTS releases will be triggered based on the number of pending of
fixes and their severity with no defect fix awaiting release more than forty-five (45) days.
 CVE fixes would trigger the immediate release of an LTS branch.

Users and developers must be able to verify that all fixes applied to an LTS are carried forward
to future releases.  Therefore, any commits backported to an LTS branch from master must merged
in a traceable manner.  To demonstrate the feasiblity of using git merge for this purpose,
I have put together a gist [1] to create traceability by backporting a change from a master
to a maint branch.

Resourcing and Proposed Timeline

Broad community support is vital to guarantee the twenty (20) month support period for each
LTS branch.  Given the ebbs and flows of contribution and committer priorities, ShapeBlue
will provide a release manager, as well as, engineering support to fill any contribution gaps
to ensure that the community fulfills LTS commitments.

In order to prepare for supporting LTS releases, we would need to complete the following items:

  1. All tools that do version number comparisons would be made aware of the LTS versioning
  2. Officially support running the management server on Java 8 since Java 7 has been EOL
since last April [1] (i.e. compile to 1.7 with the Java 1.8 compiler and run on Java 1.8).
 Providing a 20-month support window on an EOL JVM is an unacceptable security risk.
  3. Update the wiki and website to explain the new release cycle and help users decide which
release type suits their needs
  4. Define an initial test plan for LTS stabilization
  5. Agree on the definitions of ticket severities

In order to address these items and start on a regular rhythm, I propose that first LTS cycle
begin on 1 July 2016.  In the interim, we would continue to ship critical bug fixes from the
4.5 branch as it seems to be a defacto LTS branch by our users.

Together, the monthly and LTS release cycles address the needs of the entire Apache CloudStack
user community.  I believe that the process described in the proposal will yield releases
that meet the needs of users requiring release stability without adversely affecting the velocity
of the monthly release cycle.


## END ##

If no significant issues are raised by COB Friday, 5 Feb 2016, I will open a vote on this
proposal Monday, 8 Feb 2015 with the hope of gain agreement on an LTS release cycle by the
end of next week (12 Feb 2016).

Thank you again for the great feedback — I think it has greatly improved the proposal,

> On Jan 20, 2016, at 6:56 AM, Daan Hoogland <> wrote:
> Rohit,
> I don't see any reasons beyond lack of discipline, ignorance, and laziness
> in your description. Not of an RM or other individual btw but of the
> community as a whole. In point 1 and 2 you are describing how cherry-pick
> and forward merge are actually the same amount of work. In the case of
> forward merge however we will have a merge commit describing what that work
> was and still a link in later branches to the commit.
> Point 3 is actually where we should be proactive. If they don't care we
> should tell them to create a branch against the offending commit and work
> from there.
> On Wed, Jan 20, 2016 at 11:32 AM, Rohit Yadav <>
> wrote:
>> Based on my long-time experience with maintaining and doing release work
>> on 4.3 and later 4.5, there are many reasons where backporting is needed
>> and forward merge won’t work;
>> 1. Due to high codebase changes mostly due to major refactorings, it is
>> not possible to simply cherry-pick a commit; backporting many times
>> involved writing the fix manually based on the commit diff that I wanted to
>> backport. Cherry-picking becomes impossible if anyone has changed package
>> names, file/directory paths, removed code etc.
>> 2. Forward merging will also start failing (due to same reason as above)
>> as time progress the codebase diverges due to refactoring, new code and any
>> design/architectural changes. The other issue with forward merging is that
>> it would require merging from the oldest branch to the newest, with time
>> there would be several branches (given the monthly pace) between the LTS
>> branch and the future master branch. So, in my experience backporting may
>> become necessary.
>> 3. The bugfix author may not send their fix/PR against old branches, as
>> time progresses developers won’t care much about the older LTS branch(es).
>> Wrt 4.3/4.5 at times I had to backport changes myself after failing to get
>> the original author send the fix against 4.3/4.5 branch. This is expected
>> of developers, as they contribute in our own *free* time and they may not
>> have the time, bandwidth or interest in seeing those fixes in older
>> branches.
>> For example, we’ve 4.7 and 4.8 (upcoming) where forwarding merging will
>> work for sometime, but in 14-20 months the developers may not send
>> bugfix/PRs against 4.7 and expect future RMs to merge them forward as that
>> would require both merge-conflict fixing and testing for all intermediate
>> branches 4.8 to 4.20 (assuming, monthly releases we’ll at least reach 4.20
>> in 14-20 months and I’m not sure if RMs will have time and dedication to
>> test all 12+ releases/branches ). For non-LTS branches, it may not make
>> sense to even have those bugfixes.
>> In my experience (with pseudo LTS branches, 4.3/4.5) and opinion, LTS
>> branches are going to diverge wrt master with time but they are going to
>> have a dead-end.
>> About cherry-picking, the way we’re going git commits/merge at least I’m
>> not able to follow the git history at all, it looks like a mess to me. We
>> talk about ability to trace commits through branches, but I cannot even
>> follow changes in the same branch now (say master). I personally use git
>> diff and git log -p to trace changes using differences now (in files or
>> paths/folders). Cherry-picking is not bad, if done right (always include
>> the git commit ID from where it was picked using -x -s) you can trace it.
>> Regards.
>> [image: ShapeBlue] <>
>> Rohit Yadav
>> Software Architect ,  ShapeBlue
>> d:  * | s: +44 203 603 0540* <%7C%20s:%20+44%20203%20603%200540>  |  m:
>> *+91 8826230892* <+91%208826230892>
>> e:  * | t: *
>> <>  |  w:  **
>> <>
>> a:  53 Chandos Place, Covent Garden London WC2N 4HS UK
>> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
>> Services India LLP is a company incorporated in India and is operated under
>> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
>> company incorporated in Brasil and is operated under license from Shape
>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
>> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
>> a registered trademark.
>> This email and any attachments to it may be confidential and are intended
>> solely for the use of the individual to whom it is addressed. Any views or
>> opinions expressed are solely those of the author and do not necessarily
>> represent those of Shape Blue Ltd or related companies. If you are not the
>> intended recipient of this email, you must neither take any action based
>> upon its contents, nor copy or show it to anyone. Please contact the sender
>> if you believe you have received this email in error.
>> Find out more about ShapeBlue and our range of CloudStack related services:
>> IaaS Cloud Design & Build
>> <> | CSForge – rapid
>> IaaS deployment framework <>
>> CloudStack Consulting <> | CloudStack
>> Software Engineering
>> <>
>> CloudStack Infrastructure Support
>> <> | CloudStack
>> Bootcamp Training Courses <>
> --
> Daan

Find out more about ShapeBlue and our range of CloudStack related services:
IaaS Cloud Design & Build<> |
CSForge – rapid IaaS deployment framework<>
CloudStack Consulting<> | CloudStack Software
CloudStack Infrastructure Support<>
| CloudStack Bootcamp Training Courses<>
View raw message