geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason van Zyl <>
Subject Re: no more modules for specs...
Date Sat, 16 Dec 2006 07:05:29 GMT

On 16 Dec 06, at 12:10 AM 16 Dec 06, Jason Dillon wrote:

> On Dec 15, 2006, at 7:51 PM, Jason van Zyl wrote:
>>> IMO the remote repos are for user connivence *ONLY*,
>> That is horse shit. What they contain is a product of what you  
>> people put in them. You guys for example have something that is  
>> tagged, which you released and which is not reproducible. That's  
>> not Maven's fault, that's your fault. You are hosing anyone trying  
>> to consume your stuff. And if people are doing that to you then  
>> your should irate like anyone should be who looks at your tag for  
>> 1.1.1. That's why the release plugin behaves the way it does. It  
>> screamed at you when you tried to release that 1.1.1 tag I bet and  
>> you side stepped what you should have done and did something  
>> manually.
> First off... this is *my opinion*... not sure how you can jump to  
> the conclusion that my oppinon is horse shirt... or any mammal shit  
> for that matter.  But I have been know to say that mvn is crap many  
> times before... so if you feel better stating that my opinion is  
> shit... well, then go for it.
> Second, what assurance does any project have any any given artifact  
> in the central repo will remain there asis for the foreseeable future?

It's always been our policy that artifacts that we place in the  
repository are not removed. From syncing partners like Codehaus,  
Apache, ObjectWeb, or Mortbay they have to tell us that they do not  
want us to match the source they give us. We have never culled the  

> There are are already situations where bits have been added and  
> removed (at least one which I requested to get around a sync  
> problem from ASF) and a few others which I have heard about through  
> others.

Only if the repository is diddled on the source side. We cannot  
control in all places what is done. You guys seem to manually delete  
artifacts from this side which wreaks havoc.

> There is no audit trail for central or any other popular mvn repo,  
> so any release manager with half their wits is going to think twice  
> about trusting any content which is pulled from it.

Two separate issues. People behind firewalls generally do not and  
cannot use the central repository directly but they manage ones that  
do rely on central. And they can for releases because we do not  
delete anything. Everything release be signed and that will get  
easier to do automatically so though it's not fully automated to  
check the signatures you can verify what you have. We will have a  
transition period where unsigned artifacts will be accepted but  
shortly anything coming in by any means will have to be signed and  
the metadata will carry with it a reference to its source.

>>> one critical error I believe that everyone is making is thinking  
>>> that deployed artifacts are permanent...
>> Deployed releases that we place in the repository are most  
>> certainly permanent.
> What assurance do I have that artifacts are going to exist in the  
> repo in 10 years?  20?

That is our policy and we will bolster our infrastructure but Ibiblio  
gives us the assurance that what we give them stays in perpetuity.  
That's their policy. That's why we still mirror things there even  
though we are using our own dedicated box. I also have backups from  
day one that are stored in an offsite along with my other backups.

> How about the assurance that they have not been tampered with?

Signatures that you are supposed to use for releases here since  
always. We don't enforce that yet but anything coming from Apache  
should have them.

> Digest files don't do jack... as if someone alters an artifact, and  
> I download it into an empty repo... then it all looks fine to the  
> consumer.... and its highly regular for users to nuke their repo  
> when problems happen... and problems are regular too.

It will not be long before it will simply be mandatory to have a PGP  
key sign your goods if they are going to make it into the repository.

>> Snapshots can be transient depending on the policy of the  
>> repository. What I've been told is that infrastructure is telling  
>> you to remove old versions from the repository at Apache which can  
>> have disasterous effects. Your artifacts deployed to central  
>> should not disappear so I'll go make sure that we're not deleting  
>> files that have been removed from the source. But we generally  
>> assume when we are syncing from a source that the source knows  
>> what it's doing. If you delete stuff on this end then we expect  
>> you to understand what effect that will have on your clients.
> Many users may understand, but there is always one that will not,  
> which throws off the entire system.

Which users? You as a user supplier of artifacts populating your  

> If one weak link decided to re-release version 2.0 of something  
> that effectively breaks compatibility with consumers of the  
> previous 2.0 release, then then entire system is compromised... who  
> is to say that any other artifact might just change under the covers.

There are lots of the issues we know about and are fixing, lots of  
stuff in the wiki about maven-artifact changes in 2.1. You're not  
saying anything yet we haven't heard before. That's not really  
germane to this your issues anyway. You can't even get non-SNAPSHOT  
so it wouldn't matter if it was secure, it would still be useless.

> Granted that does not happen often, but it does happen, and will  
> most certainly continue to happen.

It's happened once that was very noticeable which was the overlay of  
a binary incompatible version of commons-logging 1.0 was laid down.  
Using the release plugin helps and when we make a deployment  
transactional we can enforce not being allowed to deploy the same  
version again. So it's not like we're sitting on our laurels, we've  
seen these issues and over time we add measures.

> Maybe its possible to remove the chance of it happening from  
> central, but projects do not just depend upon central... its easy  
> enough to setup a repo, then include that into your project.  And  
> the managers of that repo may remove add/change artifacts at will.   
> So my comment about the transitive nature of all mvn repos is much  
> more general... and certainly not mean to know mvn, but more as a  
> warning that artifacts on remote repos are much more transient that  
> many people (like you) believe they are.

Then don't use those repos, or label them as snapshot repos. As far  
as Geronimo is concerned why do you need anything more then central  
as a source? Aside from your SNAPSHOT dependencies.

This will only stop when Archiva is in full effect. The only way to  
submit anything to central will be via Archiva. Any project who  
wishes to have the same stability will only take artifacts that have  
passed through and instance of Archiva. You'll know you're using an  
instance of Archiva because we'll have a wagon for doing that and it  
will be configured. It will eventually be the default. It will simply  
be the Grizzly client and Jetty using the Grizzly connector.
>>> which they are most certainly not, so we should generally always  
>>> build from source to ensure that everything is being included at  
>>> the right version.
>> No you shouldn't. That defeats the entire purpose of Maven.
> Certainly not Jason...

Yes, building from source when not required defeats the purpose of  
Maven. It is generally not required.

> and I'm surprised to hear this from you as I have heard you talk  
> about the plug-ability of build functionality as much more  
> important to the purpose of maven than the artifact remoting.  IMO  
> the plugin framework of mvn is much more important to the purpose  
> of mvn than remote artifact handling... and more so the lossy  
> artifact handling is more of a detriment to mvn than anything  
> else.  I'd like to throw the remote artifact capabilities into the  
> trash... and then we'd find a reliable and trustworthy build  
> platform in mvn.

The remote artifact handling is one of Maven's greatest strengths. It  
relies on the integrity of the repository and when everyone expect  
us, the Maven developers, to clean up their shit it's not going to  
work. If you don't want to play nice with everyone else then stick  
your repository in SVN and be cut off from everyone else. When the  
central repository gets more robust through the use of Archiva it  
will become an invaluable resource. it's far more important then you  

Projects are not accountable for stuff they put in repositories is  
what make the public space unreliable in its default mode. Maven  
entirely reliable when the repositories you use are and many people  
do maintain their own repos. That is the norm for corporate usage. If  
you also really really want to you can use the SCM Wagon which works  
off an SCM. The unreliability factor is almost always the over use of  
SNAPSHOTs and people being cavalier with repositories. Maven will  
only be able to improve this over time when the repository management  
tools come into effect. It's not a small problem to try and make this  
all work and unfortunately we've determined people are too negligent,  
lazy, and don't think it's important to maintain the repositories  
even though they have become a critical resource.

> I can't even count the number of times that mvn builds have broken  
> due to external dependency changes... no local sources changes, but  
> come in to the office the next day and the build is just fucked.

SNAPSHOTs. Nothing with a SNAPSHOT is reliable. And by that the  
"SNAPSHOT" identifier. It is only an identifier, on the remote side  
there are no artifacts with "SNAPSHOT" in them. That identifier maps  
to a timestamped version. So if you wanted reliability from an ever  
changing mass, once you had a build that worked you could write a  
plugin that passed over them all, looked up the last timestamp from  
the metadata in the repository and plug those in. You have zero  
guarantee of a working build with a SNAPSHOT. None. Never have, never  

>> You have unmaintainable and non-reproducible builds because you  
>> have such a massive number of SNAPSHOTs and you never make any  
>> incremental releases. What do you expect when you're building upon  
>> sand?
> Ya could be... mvn's snapshot mechanism is more than enough rope  
> for a team to hang them selves 10x over.  Is that the teams fault  
> or mvn's fault for handing over the tools to hang themselves?

That's your fault, SNAPSHOTs mean unstable by their very definition.  
You don't have to use SNAPSHOTs it's a convenience for change  
__within__  your project.

>   Probably a mix of the two.  Which I why I tend to try to limit  
> the usage of that garbage... but since its in mvn, and other folks  
> just assume that if its there it must be good, they tend to use it...

I don't who told you to use SNAPSHOTs but they are bad news when you  
are using an external project SNAPSHOTs. You have squat in terms of  
reliability. Use all non SNAPSHOT versions, and central and if you  
have a breakage it's guaranteed to be in your project. Period. Can't  
be anything else. The only time really bad things have happened is  
the accidental commons logging thing and when we have failures at  
places like Codehaus, and Ibiblio. We've moved to fix that by using  
Contegix, trying out S3 and trying to get more mirrors. Again, we're  
not resting on our laurels.

> but more its also that g is a different type of project that has a  
> bunch of dependencies on external projects, which are moving at  
> similar (if not faster) speeds... and IIUC this is what SNAPSHOT  
> artifacts are meant to address, though I think that the actually  
> implemented, that with inconsistent deployment of snaps, has  
> completely failed as a solution.

Then you're really one project if you're so highly coupled and maybe  
you should just put it all together. I think that's a clear  
indication of a problem.

> for example, G server depends on OpenEJB... but who knows when the  
> latest snap was deployed for OpenEJB, so to ensure that your G  
> server build actually works you *MUST* build OpenEJB from source.

No it doesn't. Once you have a build that works, you can use the  
timestamp as the version and stop letting it swing. Ask OpenEJB to  
deploy everything to one repository and have no separate SNAPSHOT  
repository so that they won't be nuked. If you cannot stop using a  
SNAPSHOT something is wrong.

> This is not a mvn problem by itself, but a process problem for not  
> getting regular snaps deployed.  But assuming that there was a  
> nightly deploy of all that stuff, then there is still a window  
> during the day when the code could have changed in OpenEJB and G  
> server which would not be picked up due to the SNAPSHOT not being  
> deployed until hours later...

Absolutely, don't use them all the time. They have to do this  
periodically to give you something stable.

> and then even once it was deployed, the default policy of waiting  
> daily before updating snapshots... basically makes a huge window of  
> times when a user might get the wrong code and end up with a failed  
> build.

No it's not. I think you have a fundamental misunderstanding about  
how versioning works in Maven. At least in Maven 2.x. OpenEJB can  
deploy as many snapshots as they like, you have found one to work.  
You have found a set of snapshots to work then lock them down.

> This only happens for a small number of deps, where there is tight  
> coupling... but it does happen... and since G server and OpenEJB  
> are so intimate with each other, we run into this all of the time.   
> And there really is no solution to solve the problem, short of  
> decoupling them, or building from source.
> And since the chances of these being decoupled anytime soon is slim  
> to none... we really have no choice but to build openejb2 as a  
> dependency.

That's simply nuts. That you have to do that means you're screwed.  
That you, at no point in time, can get a stable release is not good.  
Everyone needs to across your project must settle on a version as the  
sign post.

>> Get the producers of your artifacts to release things more  
>> frequently, this ensures that SNAPSHOTs are removed from the  
>> chain. I look in your server build and you've got SNAPSHOTS  
>> **all** over the place. OpenEJB, ActiveMQ, Tranql, your spec JARs.  
>> You control, or have relationships with most of these projects.
> Give me a break dude... we have relationships with Maven too... but  
> have had little to no luck in getting some critical bugs fixed  
> which affect our builds, like how we have to split up our build  
> into two stages to avoid the bug of extensions that are built in  
> the same cycle from being picked up.

Like the one Brett  applied last week for you? LIke the RAR plugin  
that was put into the cycle for you? Like the XMLBeans plugin I have  
tried to get released for you? LIke the plugins I'm making to try and  
make staging easier and licensing easier that Dain used yesterday to  
make your release?. I took your GPG plugin and integrated it into  
Maven plus I field the bitching and fix shit for people all the time.  
It doesn't get all fixed at once. But I deal with this shit all time  
where Maven is getting pegged for shit that the tool is not  
responsible for. There are lots of bugs because we have tons of users  
and they find shit all the time. I could give a rat's ass what anyone  
thinks because we're honestly can't do anymore then we already are.

> The same shit happens for other projects too.  G depends on a lot  
> of other projects, and its non feasible to wait for our dependency  
> projects to deploy the right versions for use to move forward all  
> the time.... otherwise we would be waiting forever for other groups  
> to move.

A timestamp will do as a version. Always has in Maven 2.x. You don't  
need a final release. You need something that will not change out  
from underneath you which a SNAPSHOT is meant to do. So you have: 

Now if you look in: 

You will see that 20061122.035631 was the actual name of the JAR that  
corresponds to the latest SNAPSHOT. That will work as a version that  
will not change. For all intents and purposes that SNAPSHOT should  
stick around long enough for you to get a release.

> For example... if we wanted for your team to fix MNG-1911 before we  
> finished our m2 conversion... well, we'd still be using m1 today...  
> and probably for the foreseeable future.

Well, what does any software project do given many demands and  
limited resources? It's got 0 votes. Scream louder. I use them to  
gauge what I do in plugins:

What else are we supposed to do with the unsatisfied multitudes?

> That... and the fact that mvn releases each damn little part of a  
> project severally forces each of these projects to go through legal  
> hell to make sure that everything is up to policy with legal files  
> etc... which normally would only have applied to source files and  
> to the finally distribution.... but mvns  opened that up to include  
> almost every single damn file that is part o a build... which is  
> ridiculous.

Modularization is a good thing. Crappy problems like the legal  
problem can be solved when the group of Maven users needs something.  
That crappy legal problem has been fixed and we're testing it and it  
worked for your release yesterday. Dain used it and you don't have to  
copy any legal crap into every JAR. It's done transparently now, or  
will be when you inherit from the ASF POM:

That one I listened to and fixed.

>> You don't wait until it's time to release to get the producers of  
>> your dependencies to crap out some junk you can use. Get them  
>> release, lock your stuff down to it, move on. Repeat. You should  
>> have no SNAPSHOTs other then ones for your own projects in your  
>> POMs. If everything is changing everywhere it's total insanity
> This is not always a possibility... if any of these groups releases  
> there goods a slow or you our we do, then nothing is going to  
> move.  Using SNAPSHOTS, or building from source is really the only  
> thing that is allowing us to move forward, because there is so many  
> changes going into each of these components which are required for  
> G to function.  Its simply not possible to lock step our progress  
> with the release schedule of other projects, we must for  
> development take incremental snapshots of code.

That's fine but you still can lock down on a timestamp. You have to  
at some point draw the line. The second you lock one thing it forces  
others to lock down. You declare that for this week using this  
version of OpenEJB, everyone must follow along and you prepare for  
the following week to use the SNAPSHOT built on X day. Everyone  
trying to look at sources to match signatures is simply not scalable.  
Locking down to a version, even if it is a timestamp, is your way of  
communicating "you're going to have to coordinate with others to  
align things and live with with for the week". Things cannot change  
so radically that everything breaks. People have to use feature  
branches or be more careful about what they are expelling. Additions  
should not affect consumers of artifacts, and if it is only additions  
(which I assume because you're all following JEE standards) then you  
have to plan that better for consumption. If people are breaking  
binary compatibility five times a week they should be tarred and  

You can use CLIRR to nail offenders:

And look at that ugly Maven site and you know there's a plugin.

>> Build everything from source is not something you should be doing  
>> because you should not have to. It is one of the most obvious  
>> signifiers that you are doing something very wrong.
> I do not meant to build everything from source... though if I did  
> feel that was needed, it is most certainly not a sign of doing  
> something wrong.  If I wanted to know exactly what changes went  
> into the system... or wanted to know what changes affected which  
> tests, then that is really the only way to tell.

Again it's not. You have crappy releases and crappy planning. You  
have too much change to absorb reliably at once and is unsustainable.

> I'm not suggesting that we want to build commons-logging or howl  
> all by hand... but for the projects that change the most (those  
> which we are using SNAPSHOTS for), then for the automated scenario  
> is it ideal ti build from source, and eliminate any guess work that  
> might come in to play when using the lossy mvn SNAPSHOT downloading  
> mechanism.

You're using the SNAPSHOT mechanism incorrectly. It's for internal  
use on a project and external dependencies a rarity.

> This is especially key, when build take a while to perform, and  
> during the time it takes to run a longer build that one or more  
> snapshot builds of a dependency are created.... unless the  
> automation system actually tracks the exact artifacts produced and  
> hands them to the target build, there is no way that any  
> correlation from build to build can be done.  This is a CRITICAL  
> flaw in the mvn SNAPSHOT mechanism.

No it's not you use it incorrectly. You always have the exact version  
number for a SNAPSHOT. Always.

> We don't need to guess what changes were include in a build... but  
> for large projects and distributed builds... mvn offers no  
> assurance to what artifacts are actually used.

You can always resolve the exact version. You guys have to many which  
would make it annoying to look in the repo to find them. But we could  
easily make a tool.

>> You have way too many points of variation and all your projects  
>> seem to be in constant swing with one another. If that's the case  
>> it begs the question whether they really separate and why not just  
>> lump them all together if they are so coupled together. Which  
>> generally the case when you cannot eliminate SNAPSHOTs from your  
>> build. When you see SNAPSHOT not from your project in our POM it  
>> is a warning sign.
> Dunno about that... the most tightly coupled SNAPSHOT we have now  
> is OpenEJB... and really, I think that its so tightly coupled that  
> it really should be in the same codebase.  BUT, that is not what  
> other key plays believe, so then we are left trying to work the  
> build system to generate something that is predictable given the  
> circumstances.  Unfortunatly, mvn does not have a good answer for  
> this problem... all it has is SNAPSHOT... and IMO that is not worth  
> a damn.

Not the way you use it because that's not what it was intended for.

> Its more trouble than it is worth... and more so since all you  
> people are using it everywhere, its very difficult to educate  
> people to its harm.

I don't think people use it the way you do. Most people seem to  
understand it's for dealing with variation on the project you are  
working on. And that SNAPSHOTs typically occur within your own reactor.

>> You guys seem to want to use the release plugin for releasing  
>> things with SNAPSHOTs in them. That's not what it was made for and  
>> again it's a function of you not managing your relationships with  
>> your dependent projects. A release with SNAPSHOTs is not a  
>> release. Tagging something that contains a POM with a SNAPSHOT  
>> identifier is completely useless. I mean look at your 1.1 release:
>> That's insane. That's a tag! How did you do a release with  
>> SNAPSHOTs? That's not Maven's fault if you can't built that again  
>> if you were careless enough to tag something with SNAPSHOTs. You  
>> have got to use releases of plugins, parent POMs, and dependences  
>> more frequently. Maven has lots of bugs, sure, but this mess is  
>> largely your own doing. There are lots of Maven users with way  
>> more source code and modules then you and don't have nearly the  
>> number of problems you do.
> I had nothing to do with any of that release... I have been trying  
> to unfuck a bunch of issues with the release process in general....  
> and for the record its not all mvn related stuff that needs  
> unfucking.  Its more policy on how one uses mvn that needs work...  
> some issues with the default plugins, some bugs, but more than  
> anything it more the policy about how one uses mvn to build/release  
> projects rather than mvn itself.

The stuff done of late for resources will help everyone at Apache.  
The legal and signing nonsense will be taken care and I will take  
measures to clean up the repository fuckups.

>>> I really feel that the little short-cuts and side-steps around  
>>> maven problems are going to kill us.  And really, I don't see how  
>>> any medium to large sized group can effectively use maven to  
>>> manage releases of their projects...
>> Lots of projects do it fine. Your dependency chain is fragile  
>> which is the problem. Your coupling is too high which will not let  
>> you do releases as you have to, or you are not working with the  
>> producers of your dependencies to create releases. In trunk right  
>> now you have a SNAPSHOT for the POM that seems to contain the  
>> dependencies for everything you need. That's wrong. You need to  
>> release those parent POMs and the dependencies in them. It is  
>> natural to have SNAPSHOTs for the project in question, for CI and  
>> while you're developing. But a SNAPSHOT in a dependency means that  
>> you, as a last resort, had to use something in flux because it  
>> cannot work otherwise. This immediately should signal you to  
>> contact that project to fix it, and release it. And again, because  
>> all your dependent projects seem to share many of the developers  
>> you should be able to do that.
> Lots of projects do fine... sure.  Smaller projects with less  
> requirement for durable/repeatable and stable builds should work  
> fine.  But larger projects, with more moving parts that need to  
> have builds repeatable over 10 years+ (with no changes to tags)  
> will almost certainly fall flat on their faces when using mvn...  
> UNLESS certain precautions and limitations are heeded.

I'm building several projects way bigger then yours for mission  
critical systems that have governments involved if they don't work  
and it builds fine, and it will build fine in 10 years.

> The same is true for almost any other build tool... make and ant  
> both can end up in the same situation... BUT... the key difference  
> is that mvn makes it way to easy to get into a situation where your  
> builds are non-durable and non-stable right out of the box.  Its  
> got instabliltiy and non-durability built right into it with its  
> remote repository... a problem which make and ant do not suffer  
> from... and a reason why ant is more prevalent in commercial shops  
> than mvn is.

That's just not true. You seem to fundamentally misunderstand the use  
of a SNAPSHOT. And a build from 3 years ago running against Ibiblio  
done as a proper release will build. In Norway more people use  
Maven :-) And more and more people are using Maven. Our stats are  
going through the roof and repository usage rate goes up all the  
time. Corporate users also manage their own repositories so they  
don't have any problems.

>  * * *
> Anyways, I'm gonna stop here.  I could certainly go on and respond  
> to everything in this mail... but I don't see the point.
> My purpose was not to diss mvn here... but more so to suggest that  
> there are more effective ways to use mvn which fit better into our  
> integration model... there are better ways to use mvn ASIS with the  
> requirements/restrictions we have now.
> But even more so... my point was... that changes that have been  
> made recently have really fucked over the work I have been doing to  
> produce automated reliable durable builds for our projects... and  
> yes, based on all those SNAPSHOTS of all those projects.
> I had it working and it was working well... until the model of  
> versioning specs changed... more specifically since the specs/trunk  
> branch could not actually build and now it builds artifacts which  
> are not being using by anything... the bits that are needed are now  
> only available to build off of little spec module branches, which  
> is a huge pain to automate.
> IMO the only problem specific to maven here.... is that this  
> versioning model was allowed by maven and even more so promoted by  
> mvn since that is  how you folks manage your plugin modules.  So we  
> have the lemmings following by example... right off the cliff.
> And lastly... none of this is anything I consider mine... or my  
> own... its all junk that I have picked up and been trying to fix.   
> And at one point I did believe that mvn was going to help me do  
> that... in retrospect... Ant would have been a better tool.... less  
> bugs... less mystery... less time trying to figure out just what  
> the fuck broke.

I doubt it. When it came down to releases and updating artifacts and  
trying to tie everything all together. Lots of people might have  
problems but even now I bet there are people who could help with the  
build if necessary. If you did it in Ant you would be the only person  
who would know how it worked. You would get the nice Inner Platform  
Effect with a build your size:

I'd bet my life you would have more overall problems using Ant. Just  
because you could get it to work doesn't mean it would scale or be  
something anyone else could comprehend. It's probably already hard  
enough with what you have.

> I cant' even count the number of times I have had to nuke my local  
> repo, rebuld specs, openejb, g by hand to debug someone else's mvn  
> problem... even more prevalent is the number of times I have heard  
> of other people doing the same.  We spend more time fucking with  
> mvn than we do actually writing code for Geronimo... and that is  
> certainly a sign that something is wrong.

Yes, there are a handful of problems with maven-artifact that cause  
some grief. Can't fix everything all at once.

> --jason

View raw message