maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Connolly <stephen.alan.conno...@gmail.com>
Subject Re: Maven and DevOps
Date Sun, 15 May 2016 11:49:59 GMT
On Saturday 14 May 2016, Hohwiller, Jörg <joerg@j-hohwiller.de> wrote:

> Hi there,
>
> thanks for all your feedback.
>
> Just to clarify:
>
> * I do not want to really discuss the processes of mojohaus or maven
> community projects with this thread. If you get inspired to simplify by
> this discussion like Tamás (thanks) even better - but mojohaus process
> discussion should of course take place on mojohaus dev at googlegroups and
> even for maven dev out of this thread.
>
> * I do not want to blame anybody or say the community did not already do a
> lot of automation. Thanks for your effort so far and all that has already
> been archieved.
>
> * I understand that what I was actually proposing is partially covered by
> releasing SNAPSHOT versions to an internal but pulically available repo.
> However, see all the discussions on the web that imply that people would
> like a feature to "relabel" an already build release (e.g. some RC5) as the
> final release without rebuilding (as nothing has to change and all qualitiy
> rounds have passed). What I want to do is challenge you to think further
> and show some potential that could be archived with some improvements in
> the maven tool ecosystem.
>
> What I wanted to say is that I see still a lot of room for improvement in
> simplification of deployment:
>
> * Instead of creating a PGP key, registering that key, getting permissions
> and passwords for deployment repository, logging into nexus after
> deployment, searching for the staging repository and closing it, voting,
> waiting, refreshing, publishing, etc. this could all be completely
> automated by a smart CI chain (what all the DevOps deployment pipelines are
> about). Deployers would then only require permission to a release branch
> (or to accept PRs) and the rest would be covered by CI. And this can
> already be done today with all the tools available. Still traditional
> versioning, (automatic) tagging, etc. could take place and good old voting
> processes (on PRs instead) could be kept. This is not what maven or
> mojohaus seems to aim at but possible.


>
So to explain. Part of the issue for how maven does things is to do with
the ASF.

When we vote on a release we are voting *on the source bundle*

You might not think it, but the binaries are not actually part of the vote
(from the foundations's point of view... Yes we care about them, but
*legally* it is the source and only the source that is voted on)

There are some additional process constraints that effectively means we
currently are not in a place where a CI system can produce releases for
us... Now *i* could set up my own CI server with my own GPG key (if I
trusted my CI server) and do releases with that CI server, but that is
about it.

I have been campaigning for a nexus feature to *add* GPG signatures to a
staging repo... Such a feature would let us have a general purpose CI sign
the artifacts and as each PMC member votes, they would add their signatures
to the source bundle... Ideally the first added signature would remove the
CI system signature or we would jut generate a new CI key for every release
(as you don't want people trusting the CI server's GPG key

Anyways, the point is for ASF hosted projects *legally* for now we have the
process we have and absence some technical changes, we are stuck with it
mostly


> In this context you should also be aware that some releases have been
> messed up because platform differences (EOL-style, etc.) and a lot of other
> things can go wrong on the machine of committers causing support effort for
> the community (still the maven site update fails all the time for me at
> mojohaus if I build on windows, also for maven-release-plugin I can only
> build releases with strange workarounds, etc. but all off-topic).
>
> * What I was actually requesting to think about for the tooling is
> probably more a matter of maven repo tools such as nexus or artifactory. So
> I guess also the wrong place here. I have seen all the great potential in
> social and open collaboration. For this reason I would like to see an
> advanced toolchain that publishes the release of every successful CI build
> with a technical "version" build from timestamp and potentially the
> revision of the code repo. That repository should then have social features
> allowing everybody to give feedback to any artefact. Then the actual
> releasing could be just a "relabelling" of the technical version to a human
> readable version. I was thinking of a human readable version just as an
> alias or "symbolic link" to a technial version.
>
> * Assume maven would habe been designed from the scratch in a way that the
> "version" is always a timestamp+revision and the human readable version
> (4.1.0.RELEASE) is just a link to that technical version (just similar to
> what is done with multiple SNAPSHOTs of the same version today)... I
> personally think this would solve all the relabelling version issue I
> stated above.
>
> * My entire vision was to be able to mark an artifact with valuable
> metadata such as taggig it as bad (insecure, flawed, or whatever) so I
> could get warnings whenever I use that artifact in my project (instead of
> OWAPS dependency-check that quadruples the build time). Also a repository
> could collect a lot of meta-information (how many downloads of each
> artifact have been done, the maven client could also send an anonymized
> hash of the groupId of the project requesting the dependency so you collect
> statistics that can tell how many projects are using a particular artifact,
> etc.). Assume you could directly see how many projects are using an open
> source artifact - just one? Or 100? 1000? Quite valuable infos...
>
> Still I hope I could put seeds to some inspiration... I do not expect
> anyone here to change maven or nexus codebase due to my comments. :)
>
> However, in my business project I leveraged a great potential with an
> entirely automated deployment pipleline (that also installs the release on
> test and staging environment after integration tests passed). Many years
> ago the Open Source teams were sometimes way ahead of what was done in
> business IT projects but this has IMHO inverted now...
>
> Cheers
>   Jörg
>
> Am 02.05.2016 um 19:06 schrieb Hervé BOUTEMY:
>
>> I don't really understand
>>
>> yes, http://www.mojohaus.org/development/performing-a-release.html is
>> heavy
>>
>> But if you look at the content, you'll see that:
>> 1. the 3 first sections are about quality expectations, not release,
>> 2. the next 2 are about preparing your environment,
>> 3. then releasing is only the 6th paragraph, with 2 minimal commands to
>> run
>> release plugin
>>
>> All the next parts are only about MojoHaus choice to have a staging (like
>> Apache), with a manual peer review process: if you choose to have another
>> process, just don't use staging and the previous use of release plugin
>> will
>> just immediately publish without review
>>
>> Like Karl Heinz pointed out, there is even the new properties to release
>> without tagging: but with such an option, tracking what updates go in a
>> new
>> release is not easy for artifact consumer.
>> I personnally prefer the SNAPSHOT approach for immediate publication of
>> any
>> commit: a release is about choosing to make a special publication for more
>> guaranteed consumption by end-users. And the extra-work is manual since
>> it is
>> about associated documentation: I really don't see what can be automated
>> in
>> this part
>>
>> Then really, I don't understand what to do: we have IMHO 2 types of
>> expectations, each one with adapted associated efforts
>> And the long documentation is just because conventions are written down
>> for
>> the whole team to share: nothing that is about release process.
>>
>> Do I miss something?
>>
>> Regards,
>>
>> Hervé
>>
>> Le lundi 2 mai 2016 09:51:59 Tamás Cservenák a écrit :
>>
>>> +1 to Jorg's comments ;)
>>>
>>> I believe what Jorg says is that process is _manual_:
>>> - do something NOW (merge PR, write email for vote, ...)
>>> - come back after 3 days (and do something more...) (what if you get a
>>> baby
>>> in between?)
>>>
>>> Ideally, all you'd need is to start a small snowball: merge the PR, and
>>> initiate a release (by filling in a form of some app?). Then, the app
>>> could
>>> spin up a vote, that once votes are there, could respond to ML about vote
>>> results, would spin up a release, deploy site, etc.... without any
>>> assistance :)
>>>
>>> And during all that you are blissfully looking at another PR :)
>>>
>>>
>>> Central as collaborative platform... is really great idea, while it is !=
>>> staging at all. Staging is merely meant as QA, not in sense Jorg
>>> explains...
>>>
>>> Thanks,
>>> T
>>>
>>> On Sun, May 1, 2016 at 11:44 PM Karl Heinz Marbaise <khmarbaise@gmx.de>
>>>
>>> wrote:
>>>
>>>> Hi Jörg,
>>>>
>>>> On 5/1/16 10:18 PM, Hohwiller, Jörg wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> I wanted to share some thoughs I had recently. Maven introduced a
>>>>> revolution to the Java world and made a really big step for dependency
>>>>> and build management. Open-Source projects are more productive with
>>>>>
>>>> maven.
>>>>
>>>> However, in the last years DevOps showed up and projects start to
>>>>> continuously build releases in some cases with a fully automated build
>>>>> pipeline.
>>>>>
>>>>>
>>>>> When I look at open-source development around I see that we have great
>>>>> infrastructure with github and pull-requests, etc.
>>>>>
>>>> Hm..Apache accepts pull request as well as MojoHaus
>>>>
>>>> But as a downside I also see slow and over-complex processes to get
>>>>> something released (see e.g.
>>>>> http://www.mojohaus.org/development/performing-a-release.html - wow
>>>>> that
>>>>> is not really lean).
>>>>>
>>>> Sorry to say that, but you haven't had the opportunity to see over
>>>> complex processes...
>>>>
>>>> Simply make a release and open the VOTE for 3 Days (with lazy
>>>> census)..what is so complex about that? Yes you need to do some things:
>>>>
>>>> make a release:
>>>>
>>>> mvn -B release:prepare release:perform (gpg signing)
>>>>
>>>> go to target/checkout
>>>>
>>>> mvn site site:stage scm-publish:publish-scm
>>>>
>>>> and send an email for VOTE and wait the result and than click on the
>>>> release button for publishing the artifacts to Central...
>>>>
>>>> Hm..?
>>>>
>>>>
>>>> Furthermore if you don't like it than simply start a VOTE to change
>>>> that...What exactly is the problem herer?
>>>>
>>>> I think that those three days are a good trade off between speed and the
>>>> options for developers to react cause not all developers are sitting in
>>>> front of the keyboards and just waiting someone will start a VOTE (they
>>>> are doing other things as well)...
>>>>
>>>> In order not to fingerpoint someone I will pick myself: I got a
>>>>> pull-request from someone for servicedocgen-maven-plugin that I
>>>>> maintain
>>>>> at mojohaus. I reviewed the PR and merged it. Unfortunately, I was very
>>>>> busy then and did not create a release for two month now. It might not
>>>>> take that much, but still too much. I want to question why do we need
>>>>> all this stuff and the votings, etc.
>>>>>
>>>> Because the community which means the Devs of MojoHaus have decided to
>>>> do so...(I'm a member of this as well)...
>>>>
>>>> BTW: Clearly saying to discuss things here on Apache Maven Dev list
>>>> which belong to MojoHaus is not a good idea...
>>>>
>>>> For apache the situation is a little bit different, cause there are some
>>>> other aspects important...(Like legals etc.)...but no one prevents you
>>>> from starting a release....
>>>>
>>>>
>>>> And yes sometimes mojohaus/apache maven devs are busy...so those are
>>>> open source projects ? I don't understand the remarks ?
>>>>
>>>> So assume the following future vision for a maven project:
>>>>>
>>>>> * When a pull request passed (travis, coveralls, etc.) and gets merged
>>>>> a
>>>>> CI system automatically builds a release (no need to get PGP keys per
>>>>> developer just one setup once for the project CI). The release simply
>>>>> gets a timestamp as version-identifier (yyyyMMdd-hhmmss).
>>>>>
>>>> Using PGP etc. is very important in particular related to discussions
>>>> about trust and security...(of course only two aspects of that)...
>>>>
>>>> You can remember a feew weeks ago as a single NPM developer could remove
>>>> artifacts from NPM which broke thousands of builds? This showed me that
>>>> the NPM people don't want to learn from the experiences of Maven central
>>>> already had....
>>>>
>>>> Apart from that this would produces a release per commit which usually
>>>> not a good idea...CI/CD is the opportunity to make every state a release
>>>> ...So furthermore you can use staging in repository managers like Nexus
>>>> to handle such things?
>>>>
>>>> * Now besides the project being responsible for quality (by having good
>>>>> tests and only accepting PRs after reasoable review) the community
>>>>> (artifact users) could also help and do additional quality assurance.
>>>>>
>>>> This contradicts to the previous suggestions...
>>>>
>>>> The thing you mentioned...the community is important and must give
>>>> feedback which is not always the case or not in the number we wished it
>>>> should be....
>>>>
>>>> Assume maven-central would become a collaborative platform where the
>>>>> users of artifacts could vote and label artifact releases. Add
>>>>> comments,
>>>>> link CVE or bug reports, etc. Vote +1/-1 on quality or security...
>>>>>
>>>> This is more or less already done via the staging before going to Maven
>>>> Central...this is what the 72 hours VOTE times is for...
>>>>
>>>> Why should they put that on central...the projects are responsible...
>>>> Yes you could imagine this ...but in real life i don't see it...cause
>>>> someone needs to maintain the platform Central in this way...who should
>>>> do that?
>>>>
>>>> I have large doubts that a users will vote on Central for a release
>>>> ...cause they are blaming things which do not work (see for example
>>>> StackOverFlow) but not filling a JIRA or at least mail on the mailing
>>>> list etc.
>>>>
>>>> * Still the project releasing the artifacts could label releases and
>>>>> associate minor/major release numbers to branches.
>>>>>
>>>> Why should we/others associate minor/major release numbers to branches?
>>>> Which advantages should that give ?
>>>>
>>>> SemVer.org shows better ideas from the point of view what the artifacts
>>>> are doing for users......
>>>>
>>>> In such case however dependencies would not point to a version like
>>>>> (4.2.1.RELEASE) but instead to 20160501-235901. In order to pick the
>>>>> right technical version you would lookup the collaborative meta data.
>>>>> I do not expect everybody to shout hurray to this rought idea. But I
>>>>> would be happy if people can think about it and may combine it with
>>>>> other ideas so we get even better in the future some day.
>>>>>
>>>> Why should i use something weird like 20160401-232211/20160401-232212
>>>> instead of  1.0.0/1.0.1 ? Where is the difference ? What does this mean?
>>>>
>>>> I don't see advantages in comparsion to 1.0.0 / 1.0.1 more the
>>>> contrary...
>>>>
>>>> See also
>>>>> https://dzone.com/articles/continuous-releasing-maven
>>>>>
>>>> If we are talking about CI/Cd than using release plugin might not be the
>>>> best choice...than using versions-maven-plugin etc. are better
>>>> alternatives..Furthermore automatic merging in Jenkins etc. can be done
>>>> as well...and creating a release can be done within a pipeline without
>>>> any problems...
>>>>
>>>> https://devopsnet.com/2012/02/21/continuous-delivery-using-maven/
>>>>>
>>>> If you have taken a deeper look during the last months...there are
>>>> several activities to make the life cycle more flexible in different
>>>> ways...or to be more accurate to make the way open to change things...
>>>>
>>>> For example:
>>>> https://issues.apache.org/jira/browse/MNG-5697
>>>>
>>>> I would love to see that maven better supports flexible handling of the
>>>>> version for the development
>>>>>
>>>>>   > view while it could simply stay as it is for
>>>>
>>>>> the consumer view. When I use variables in versions of POMs (and I do
>>>>> that in every project today e.g. in combination with
>>>>> flatten-maven-plugin) I always get warnings from Maven saying:
>>>>> [WARNING] Some problems were encountered while building the effective
>>>>> model for ...
>>>>> [WARNING] 'version' contains an expression but should be a constant.
>>>>>
>>>> Than you are using a wrong Maven version...starting from Maven 3.2.1 you
>>>> can use things like this in your version:
>>>>
>>>> from the release notes:
>>>> http://maven.apache.org/docs/3.2.1/release-notes.html
>>>> <quote>
>>>> A simple change to prevent Maven from emitting warnings about versions
>>>> with property expressions. Allowed property expressions in versions
>>>> include ${revision}, ${changelist}, and ${sha1}. These properties can be
>>>> set externally, but eventually a mechanism will be created in Maven
>>>> where these properties can be injected in a standard way. For example
>>>> you may want to glean the current Git revision and inject that value
>>>> into ${sha1}. This is by no means a complete solution for continuous
>>>> delivery but is a step in the right direction.
>>>> </quote>
>>>>
>>>> So you can simply set the version like:
>>>>
>>>> <version>${revision}</version>
>>>>
>>>> and than you do:
>>>>
>>>> mvn -Drevision=1.0 deploy
>>>>
>>>> or
>>>>
>>>> mvn -Drevision=1.0-SNAPSHOT deploy
>>>>
>>>> So where is the problem here?
>>>>
>>>> Still I see no clear picture how maven will adress these needs that
>>>>> obviously many developer seem to have.
>>>>>
>>>> What needs have many developers..?
>>>>
>>>> We could start a discussions about that...and see if we get a clear
>>>> picture where the journey could go...
>>>>
>>>> Issues like MNG-4161, MNG-624 and others were simply closed and not
>>>>> fixed.
>>>>>
>>>> MNG-624 would lead to using a version range like JavaScript (NPM) does
>>>> etc. and based on the experience we had on version ranges....Furthermore
>>>> in the meantime you could use version ranges in parent with next Maven
>>>> Release (3.4.0) see
>>>>
>>>>
>>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20MNG%20AND%20fix
>>>> Version%20%3D%203.4.0
>>>>
>>>>
>>>> MNG-4161: This can more or less being solved by using
>>>> dependencyManagement in parent of the multi module build...Yes not the
>>>> optimal solution...but others would need changing the fundamentals of
>>>> Maven...and somethings needed to be changed in the pom modell..
>>>>
>>>> Apart from that Benson Margulies has something via Extension: (User
>>>> Mailing list):
>>>>
>>>> https://github.com/basis-technology-corp/auto-version-maven-extension
>>>>
>>>>
>>>> Kind regards
>>>> Karl Heinz
>>>>
>>>>   > After just seeing that and levaing an angry comment, I was about
>>>>
>>>>> to cancel this mail. However, I just decided to still send it but have
>>>>> little expectation that it will make any sense...
>>>>>
>>>>> Best regards
>>>>>
>>>>>    Jörg
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

-- 
Sent from my phone

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message