Mailing-List: contact dev-help@aries.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aries.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=DKIM-Signature:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer;
  b=PUBGC/xqlpLRB5288XYg1MOjkuI8Kzg6zVAlTPbgC0EMKCK9pjXA5TNoJk958Ftt5kUNH/iFMiwF+ZZJ7MZxLU7/0Rm1x0dQKZH6GblZ8ADl4pS6NU3eqmsnJnEfTOYbkndHZSRm/z+DJh1dgasKYVRiuLr7kuF7++i3mc1JiIA=
  ;
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: [DISCUSSION] Modifications to Aries release process
From: David Jencks <david_jencks@yahoo.com>
In-Reply-To: 
 <CAC9tjn9BWJte3R0-RMAqXP=WCG5CbwNbUeCEzVg6738+LbrJ5A@mail.gmail.com>
Date: Fri, 6 Jul 2012 10:21:48 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <26DE97BB-2756-4F90-98D3-C3578EED7BE2@yahoo.com>
References: 
 <CAC9tjn90Nh3oWAzrgMindNYTQzpWvDPud008T2QRXyCT4YchKQ@mail.gmail.com>
 <2956591.HBzuJNjzLo@dilbert.dankulp.com>
 <CAEQV+EGDNabghBmgVzKqZHm1-_ELMhvoFbWSj8CkbhMMk3Svwg@mail.gmail.com>
 <CAC9tjn-bj7aE-05xy48ge6XVRKF63ZLDY1DmyKOXo3PHdVkvhA@mail.gmail.com>
 <CAEQV+EEudcUoEgRE2-sUgoxOW_rX_uYH3e_S1g1=B=rfkO7SWQ@mail.gmail.com>
 <CAC9tjn9BWJte3R0-RMAqXP=WCG5CbwNbUeCEzVg6738+LbrJ5A@mail.gmail.com>
To: dev@aries.apache.org


On Jul 6, 2012, at 9:24 AM, Holly Cummins wrote:

> Re-opening a slightly old conversation, just to ensure we have a
> record of the advantages and disadvantages of each approach to
> releasing groups of bundles. (I'm also updating the 'Releasing Aries'
> web page with the pros and cons of each way of doing it.)
>=20
> On Thu, Jun 28, 2012 at 6:12 PM, Guillaume Nodet <gnodet@gmail.com> =
wrote:
>> I think you made a wrong assumption which is that the staging repo =
has
>> to be backed by a single svn revision (which is btw not really the
>> case because it comes from several tags anyway).
>> What I'd suggest is doing the api bundles release and have them
>> uploaded to nexus, then upgrade the implementations  and all other
>> components to use those releases, commit,  release those other
>> projects, THEN, close the staging repo and call for a vote.
>> So we'd have a single staging repo / vote with a set of testable =
bundles in it.
>=20
> This approach will break trunk unless we do an extra step.
>=20
> Say we release an API bundle, at version 1.0.0. The release plugin
> will change the version of the API bundle in trunk to be
> 1.0.1-SNAPSHOT. Normally, you'd just upgrade everything to depend on
> 1.0.1-SNAPSHOT of the API bundle (and if we weren't doing release by
> bundle, the release plugin would automatically update the dependencies
> for us). However, it's important that things which depend on the API
> bundle continue depending on 1.0.0, unless there's a good reason to
> change (like they use a new method). Otherwise, we lose a lot of the
> benefits of semantic versioning. Why? If we add a new method to the
> API, the package version changes to 1.1, and anything compiling
> against 1.0.1-SNAPSHOT won't resolve against 1.0.0, even if it would
> actually *work* fine against 1.0.0. In order to avoid restricting our
> modularity, we need to depend on 1.0.0 unless the 1.0.1-SNAPSHOT
> methods are actually being used.
>=20
> So, continuing the example, in trunk, as part of the release, we've
> switched to depending on 1.0.0 of the API. This compiles in the tagged
> release, as long as bundles are compiled in a specific order. In
> trunk, it won't compile at all, because nothing in trunk is building
> API-1.0.0, and version 1.0.0 isn't in a maven repository yet. Instead,
> trunk is building 1.0.1-SNAPSHOT. So to patch trunk up, we need to add
> an extra step in which we switch back to depending on 1.0.0-SNAPSHOT,
> and then a second extra step once the release is promoted, to depend
> on 1.0.0.

I don't understand the problem.  I thought Guillaume was suggesting that =
you put a lot of released artifacts, built by the rm (you) in the same =
nexus staging repo.  Since you just built API-1.0.0 on your build =
machine, it will be in your local maven repo, and you can continue to =
build all the other stuff that needs it on your repo.  No one else can =
get the 1.0.0 artifacts from a remote maven  repo until you close the =
staging repo, but they can also build the 1.0.0 artifacts themselves, =
just like you did (they won't have to change the versions, since the =
relevant code will already be in a svn tag from running the release =
plugin).

Once you close the staging repo, people can add it to their local nexus  =
or settings.xml and fetch the under-vote 1.0.0 artifacts without =
building them themselves.

What am I missing?

thanks
david jencks


>=20
> I think this is sensible to do occasionally, but it's error prone and
> makes more work for the release manager. Obviously, the big advantage
> of it is that the release gets voted through over a period of a few
> days, rather than a month or two, for a big release. :)
>=20
> Holly
>=20
>=20
>=20
>> On Thu, Jun 28, 2012 at 6:45 PM, Holly Cummins
>> <holly.k.cummins@googlemail.com> wrote:
>>> Hi Guillaume,
>>>=20
>>> Thanks for your comments. Here are some of my thoughts ...
>>>=20
>>> On Thu, Jun 28, 2012 at 3:19 PM, Guillaume Nodet <gnodet@gmail.com> =
wrote:
>>>> I think one problem, considering the current votes, is that it's
>>>> really difficult to test anything.   Releasing api bundles with no
>>>> implementation to test is definitely not helping imo.
>>>=20
>>> I know what you mean about the testing, and I'm not totally sure =
what
>>> the best answer is. I know what I'm releasing comes from trunk, and =
is
>>> being tested by the Jenkins builds, so I'm pretty confident it works
>>> in a real system. However being tested in more environments and by
>>> more systems is obviously a Good Thing.  I think the best way to =
test
>>> the API bundles is with the current -SNAPSHOT bundles of the
>>> implementation, either in something like the blog sample or some =
other
>>> working system. If we weren't moving from 0.x to 1.0 you could also
>>> test micro releases alongside existing impl bundles to ensure
>>> everything resolves and works as claimed.
>>>=20
>>>> Holly, just a question: is there a specific reason why are you =
doing
>>>> the release in multiple votes ? It would be simpler to just release
>>>> everything in one go and wait for a longer time because there are =
more
>>>> things to check, or at least, release the api + implementation so =
that
>>>> we can actually try something.  Just my 2 cents.
>>>=20
>>> I agree that this sort of 'extended incremental' release is a bit
>>> awkward, and I was wondering when someone would ask what on earth I
>>> was doing :). IMO it's the cleanest way to work with with
>>> release-by-bundle (which I know you disagree with). If I release
>>> everything in one go, there's a problem with the dependencies =
between
>>> bundles. At the moment in trunk, almost every dependency is a =
SNAPSHOT
>>> dependency. In the past we've updated all bundles to use =
non-SNAPSHOT
>>> (but not yet released) versions in a branch, and I could even do
>>> something similar without using a branch by briefly having the trunk
>>> builds produce 1.0.0 artefacts. However, I think this creates a
>>> greater burden for testers. If there are compilation-order
>>> dependencies between parts of a release which don't share a =
top-level
>>> pom, everyone verifying a script has to compile them in the right
>>> order. I count 118 bundles to release, so that's a lot of bundles to
>>> get in the right order, and I didn't think any PMC member would want
>>> to try. :) I guess this could be automated with a verification =
script
>>> which either hardcodes or calculates the dependency graph, but it
>>> seemed to me like more work for everyone and more risk for the
>>> release. My hope was that if verifying individual mini-releases was
>>> easy enough, doing multiple ones wouldn't be a problem (and in fact
>>> would nicely distribute any effort, making it easier to vote).
>>>=20
>>> I know at this stage some of you are thinking "and *this* is why
>>> release by bundle is a bad idea!", and that's not really a debate I
>>> want to re-open. Among other things, I think any re-engineering of =
our
>>> poms at this stage will further delay the release.
>>>=20
>>> The good news is I believe this problem will almost entirely go away
>>> for 1.0.x and 1.x releases, because the impl bundle will, in most
>>> cases, depend on an *already* released version of its API bundle or
>>> another Aries component. This means a bunch of related bundles could
>>> be released at the same time, without compile issues, or a =
meaningful
>>> release really could consist of just a single bundle. That's true
>>> modularity and it should give both us and our users big benefits.
>>>=20
>>>=20
>>>>=20
>>>> On Mon, Jun 25, 2012 at 8:07 PM, Daniel Kulp <dkulp@apache.org> =
wrote:
>>>>>=20
>>>>> Honestly, with the change to using Nexus, the SHA1 and MD5 checks =
are
>>>>> completely pointless.   Nexus generates them itself based on =
what's
>>>>> uploaded.  The "is it a valid signature" part of the GPG testing =
is also
>>>>> pointless as Nexus won't let you close the repo unless the =
signatures are
>>>>> valid.   The only check you really need to do is to make sure the =
key that
>>>>> was used is "trusted" by you.   (aka: was it really Holly who =
deployed those
>>>>> artifacts)    So the monontonous parts of checking that stuff is =
really
>>>>> irrelevant at this point.  (providing we trust that infra has =
Nexus
>>>>> sufficiently locked down and secure)
>>>>>=20
>>>>>=20
>>>>> I actually don't have a big issue with the difficulting in getting =
votes.
>>>>> I'm involved in another community that has a PMC that is easily 4 =
times the
>>>>> size of this one, yet we still have difficulting getting votes =
there.
>>>>> While not ideal, life events can cause priority shifts and such so =
people
>>>>> may not be able to be as responsive.
>>>>>=20
>>>>> My bigger problem is that the entire per bundle release process =
and symantic
>>>>> versioning crap has put a HUGE burden on the release manager.   =
That makes
>>>>> it much harder to get quality releases out and makes it less =
likely that
>>>>> anyone will step up to get "minor fixes" released.   The only =
reason I
>>>>> stepped up with the 0.3.1 bp stuff is that *MY*  customers are =
being
>>>>> affected by it.   Like wise for the proxy stuff.   If *my* =
customers were
>>>>> not affected, I don't think I would have spent the time and =
effort.    If
>>>>> the process for getting fixes and releases out to users was =
smaller and
>>>>> easier, I have no problem doing them.   For CXF, we do full =
releases on 3
>>>>> branches every other month or so.   But that's because it's EASY =
to do.
>>>>>=20
>>>>> If it was up to me, I'd toss out the entire versioning thing with =
1.0 and go
>>>>> back to per module versioning thing.   So my fix to proxy would  =
have
>>>>> involved checking out all of "proxy", fixing it, and releasing all =
of proxy
>>>>> as a proxy "0.3.1", even the modules that haven't changed.   It's =
just a
>>>>> huge hassle to track down which bundles have changed, which =
haven't which
>>>>> version numbers need to be updated, etc....   If it's not quick =
and easy to
>>>>> do releases as a release manager, very few people are going to =
step up to do
>>>>> it.     It may not be 100% "proper OSGi", but IMO, getting fixes =
and such to
>>>>> the users is more important than that.    But that's my opinion.
>>>>>=20
>>>>>=20
>>>>> Dan
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> On Saturday, June 23, 2012 03:27:07 PM Holly Cummins wrote:
>>>>>> Hi all,
>>>>>>=20
>>>>>> Now that Jeremy's taken the time to write up our release =
verification
>>>>>> process, I'd like to propose we change it. :) I think it's too =
onerous
>>>>>> on the pmc, which therefore also inhibits our ability to be =
responsive
>>>>>> to our users.
>>>>>>=20
>>>>>>=20
>>>>>> ------------------------------- Why what we have isn't working =
for the
>>>>>> community -------------------------------
>>>>>>=20
>>>>>> I believe our users would like more frequent releases. We've had
>>>>>> several keen requests and tweets and comments on the aries-user
>>>>>> mailing list wishing we'd release more often. For example:
>>>>>>=20
>>>>>> * "Desperately waiting for an Aries release after loooong time.."
>>>>>> * "The problem with Aries is they seem to be too busy coding to
>>>>>> release anything."
>>>>>> * "Compared to other projects (like Karaf and Camel) Aries =
releases
>>>>>> tend to take quite some time."
>>>>>> * "It's 2012 now and Aries 0.3 is almost a year old. Is there any
>>>>>> chance of a new Aries JPA release any time soon? "
>>>>>> * "Looks like Apache Aries has made no visible progress since Jan
>>>>>> 2011, if the time stamps on the maven central artefacts are to be
>>>>>> believed."
>>>>>>=20
>>>>>> ------------------------------- Why what we have isn't working =
for us
>>>>>> -------------------------------
>>>>>>=20
>>>>>> Both Dan and I are trying to do releases at the moment, and =
struggling
>>>>>> to get enough PMC votes. Dan's release is to back port a =
show-stopper
>>>>>> proxy fix, so a release there is particularly pressing - he's got =
a
>>>>>> non-binding +infinity vote, but that's all. My test support =
release
>>>>>> vote has been open for about 64 hours, and only got one vote so =
far
>>>>>> (thanks David B!). Obviously testsupport is less exciting than =
proxy,
>>>>>> but that bundle does block more interesting releases.
>>>>>>=20
>>>>>> Why aren't people voting? My guess is that it's too much work to =
do
>>>>>> the full set of verifications described at
>>>>>> http://aries.apache.org/development/verifyingrelease.html. There =
are
>>>>>> seven steps, and while they don't actually take that long to =
complete,
>>>>>> it's enough of a burden that we tend to leave the voting to =
someone
>>>>>> else unless we really care about a release. I'm as guilty of this =
as
>>>>>> anyone - I think a release is a good idea, but I'm spending =
enough
>>>>>> time working on the 1.0.0 release that I don't want to take time =
out
>>>>>> to vote on another release. I suspect Dan might feel exactly the =
same
>>>>>> about my 1.0.0 bundles. :)
>>>>>>=20
>>>>>> With release-by-bundle, there's a lot of verifications. Excluding =
the
>>>>>> sandbox code, we have 123 bundles to release in 1.0.0. At three =
votes
>>>>>> per bundle, that means the PMC need to do 369 MD5 checks, 369 PGP
>>>>>> checks, 369 RAT checks, and so on, just to get 1.0.0 out the =
door.
>>>>>> This just doesn't seem like it scales. Batching the bundle =
releases
>>>>>> together eases some of this burden, but not all.
>>>>>>=20
>>>>>> ------------------------------- What I propose
>>>>>> -------------------------------
>>>>>>=20
>>>>>> I suggest we move to a more trust-based system, where PMC members
>>>>>> carefully check releases if they want, but where in general =
they're
>>>>>> voting on the principle of the release, rather than the mechanics =
of
>>>>>> the archives. In particular, they don't feel compelled to do =
checks
>>>>>> before voting. If PMC members could say "Our users need this =
function,
>>>>>> so +1", or "I know Holly has done sensible things in the past, so =
+1"
>>>>>> or even "Do I want to check the SHAs on a test support bundle? =
Really?
>>>>>> +1" it would get our releases moving better, and also save work =
for
>>>>>> all of us.
>>>>>>=20
>>>>>> (At the moment I think what's happening is people are thinking =
"Do I
>>>>>> want to check the SHAs on a test support bundle? Really?" and =
then
>>>>>> skipping the +1 bit. :)  )
>>>>>>=20
>>>>>> To ensure that at least *someone* has run the checks, the release
>>>>>> manager could include the output of the seven checks in an email =
to
>>>>>> the list. I think this level of checking is perfectly compatible =
with
>>>>>> the minimum Apache process, which is that the release manager =
signs
>>>>>> the artefacts and three PMC members vote +1
>>>>>> (http://www.apache.org/dev/release-publishing.html#voted).
>>>>>>=20
>>>>>> What do people think?
>>>>>>=20
>>>>>> Holly
>>>>> --
>>>>> Daniel Kulp
>>>>> dkulp@apache.org - http://dankulp.com/blog
>>>>> Talend Community Coder - http://coders.talend.com
>>>>=20
>>>>=20
>>>>=20
>>>> --
>>>> ------------------------
>>>> Guillaume Nodet
>>>> ------------------------
>>>> Blog: http://gnodet.blogspot.com/
>>>> ------------------------
>>>> FuseSource, Integration everywhere
>>>> http://fusesource.com
>>=20
>>=20
>>=20
>> --
>> ------------------------
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> FuseSource, Integration everywhere
>> http://fusesource.com