geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Dillon <ja...@planet57.com>
Subject Re: no more modules for specs...
Date Sat, 16 Dec 2006 05:10:46 GMT
On Dec 15, 2006, at 7:51 PM, Jason van Zyl wrote:
>> IMO the remote repos are for user connivence *ONLY*,
>
> That is horse shit. What they contain is a product of what you  
> people put in them. You guys for example have something that is  
> tagged, which you released and which is not reproducible. That's  
> not Maven's fault, that's your fault. You are hosing anyone trying  
> to consume your stuff. And if people are doing that to you then  
> your should irate like anyone should be who looks at your tag for  
> 1.1.1. That's why the release plugin behaves the way it does. It  
> screamed at you when you tried to release that 1.1.1 tag I bet and  
> you side stepped what you should have done and did something manually.

First off... this is *my opinion*... not sure how you can jump to the  
conclusion that my oppinon is horse shirt... or any mammal shit for  
that matter.  But I have been know to say that mvn is crap many times  
before... so if you feel better stating that my opinion is shit...  
well, then go for it.

Second, what assurance does any project have any any given artifact  
in the central repo will remain there asis for the foreseeable  
future?  There are are already situations where bits have been added  
and removed (at least one which I requested to get around a sync  
problem from ASF) and a few others which I have heard about through  
others.

There is no audit trail for central or any other popular mvn repo, so  
any release manager with half their wits is going to think twice  
about trusting any content which is pulled from it.


>> one critical error I believe that everyone is making is thinking  
>> that deployed artifacts are permanent...
>
> Deployed releases that we place in the repository are most  
> certainly permanent.

What assurance do I have that artifacts are going to exist in the  
repo in 10 years?  20?  How about the assurance that they have not  
been tampered with?  Digest files don't do jack... as if someone  
alters an artifact, and I download it into an empty repo... then it  
all looks fine to the consumer.... and its highly regular for users  
to nuke their repo when problems happen... and problems are regular too.


> Snapshots can be transient depending on the policy of the  
> repository. What I've been told is that infrastructure is telling  
> you to remove old versions from the repository at Apache which can  
> have disasterous effects. Your artifacts deployed to central should  
> not disappear so I'll go make sure that we're not deleting files  
> that have been removed from the source. But we generally assume  
> when we are syncing from a source that the source knows what it's  
> doing. If you delete stuff on this end then we expect you to  
> understand what effect that will have on your clients.

Many users may understand, but there is always one that will not,  
which throws off the entire system.

If one weak link decided to re-release version 2.0 of something that  
effectively breaks compatibility with consumers of the previous 2.0  
release, then then entire system is compromised... who is to say that  
any other artifact might just change under the covers.

Granted that does not happen often, but it does happen, and will most  
certainly continue to happen.  Maybe its possible to remove the  
chance of it happening from central, but projects do not just depend  
upon central... its easy enough to setup a repo, then include that  
into your project.  And the managers of that repo may remove add/ 
change artifacts at will.  So my comment about the transitive nature  
of all mvn repos is much more general... and certainly not mean to  
know mvn, but more as a warning that artifacts on remote repos are  
much more transient that many people (like you) believe they are.

>> which they are most certainly not, so we should generally always  
>> build from source to ensure that everything is being included at  
>> the right version.
>
> No you shouldn't. That defeats the entire purpose of Maven.

Certainly not Jason... and I'm surprised to hear this from you as I  
have heard you talk about the plug-ability of build functionality as  
much more important to the purpose of maven than the artifact  
remoting.  IMO the plugin framework of mvn is much more important to  
the purpose of mvn than remote artifact handling... and more so the  
lossy artifact handling is more of a detriment to mvn than anything  
else.  I'd like to throw the remote artifact capabilities into the  
trash... and then we'd find a reliable and trustworthy build platform  
in mvn.

I can't even count the number of times that mvn builds have broken  
due to external dependency changes... no local sources changes, but  
come in to the office the next day and the build is just fucked.


> You have unmaintainable and non-reproducible builds because you  
> have such a massive number of SNAPSHOTs and you never make any  
> incremental releases. What do you expect when you're building upon  
> sand?

Ya could be... mvn's snapshot mechanism is more than enough rope for  
a team to hang them selves 10x over.  Is that the teams fault or  
mvn's fault for handing over the tools to hang themselves?  Probably  
a mix of the two.  Which I why I tend to try to limit the usage of  
that garbage... but since its in mvn, and other folks just assume  
that if its there it must be good, they tend to use it...

but more its also that g is a different type of project that has a  
bunch of dependencies on external projects, which are moving at  
similar (if not faster) speeds... and IIUC this is what SNAPSHOT  
artifacts are meant to address, though I think that the actually  
implemented, that with inconsistent deployment of snaps, has  
completely failed as a solution.

for example, G server depends on OpenEJB... but who knows when the  
latest snap was deployed for OpenEJB, so to ensure that your G server  
build actually works you *MUST* build OpenEJB from source.

This is not a mvn problem by itself, but a process problem for not  
getting regular snaps deployed.  But assuming that there was a  
nightly deploy of all that stuff, then there is still a window during  
the day when the code could have changed in OpenEJB and G server  
which would not be picked up due to the SNAPSHOT not being deployed  
until hours later... and then even once it was deployed, the default  
policy of waiting daily before updating snapshots... basically makes  
a huge window of times when a user might get the wrong code and end  
up with a failed build.

THE ONLY WAY TO SOLVE THIS IS TO BUILD IT LOCALLY.

This only happens for a small number of deps, where there is tight  
coupling... but it does happen... and since G server and OpenEJB are  
so intimate with each other, we run into this all of the time.  And  
there really is no solution to solve the problem, short of decoupling  
them, or building from source.

And since the chances of these being decoupled anytime soon is slim  
to none... we really have no choice but to build openejb2 as a  
dependency.


> Get the producers of your artifacts to release things more  
> frequently, this ensures that SNAPSHOTs are removed from the chain.  
> I look in your server build and you've got SNAPSHOTS **all** over  
> the place. OpenEJB, ActiveMQ, Tranql, your spec JARs. You control,  
> or have relationships with most of these projects.

Give me a break dude... we have relationships with Maven too... but  
have had little to no luck in getting some critical bugs fixed which  
affect our builds, like how we have to split up our build into two  
stages to avoid the bug of extensions that are built in the same  
cycle from being picked up.

The same shit happens for other projects too.  G depends on a lot of  
other projects, and its non feasible to wait for our dependency  
projects to deploy the right versions for use to move forward all the  
time.... otherwise we would be waiting forever for other groups to move.

For example... if we wanted for your team to fix MNG-1911 before we  
finished our m2 conversion... well, we'd still be using m1 today...  
and probably for the foreseeable future.

That... and the fact that mvn releases each damn little part of a  
project severally forces each of these projects to go through legal  
hell to make sure that everything is up to policy with legal files  
etc... which normally would only have applied to source files and to  
the finally distribution.... but mvns  opened that up to include  
almost every single damn file that is part o a build... which is  
ridiculous.


> You don't wait until it's time to release to get the producers of  
> your dependencies to crap out some junk you can use. Get them  
> release, lock your stuff down to it, move on. Repeat. You should  
> have no SNAPSHOTs other then ones for your own projects in your  
> POMs. If everything is changing everywhere it's total insanity

This is not always a possibility... if any of these groups releases  
there goods a slow or you our we do, then nothing is going to move.   
Using SNAPSHOTS, or building from source is really the only thing  
that is allowing us to move forward, because there is so many changes  
going into each of these components which are required for G to  
function.  Its simply not possible to lock step our progress with the  
release schedule of other projects, we must for development take  
incremental snapshots of code.


> Build everything from source is not something you should be doing  
> because you should not have to. It is one of the most obvious  
> signifiers that you are doing something very wrong.

I do not meant to build everything from source... though if I did  
feel that was needed, it is most certainly not a sign of doing  
something wrong.  If I wanted to know exactly what changes went into  
the system... or wanted to know what changes affected which tests,  
then that is really the only way to tell.

I'm not suggesting that we want to build commons-logging or howl all  
by hand... but for the projects that change the most (those which we  
are using SNAPSHOTS for), then for the automated scenario is it ideal  
ti build from source, and eliminate any guess work that might come in  
to play when using the lossy mvn SNAPSHOT downloading mechanism.

This is especially key, when build take a while to perform, and  
during the time it takes to run a longer build that one or more  
snapshot builds of a dependency are created.... unless the automation  
system actually tracks the exact artifacts produced and hands them to  
the target build, there is no way that any correlation from build to  
build can be done.  This is a CRITICAL flaw in the mvn SNAPSHOT  
mechanism.  We don't need to guess what changes were include in a  
build... but for large projects and distributed builds... mvn offers  
no assurance to what artifacts are actually used.


> You have way too many points of variation and all your projects  
> seem to be in constant swing with one another. If that's the case  
> it begs the question whether they really separate and why not just  
> lump them all together if they are so coupled together. Which  
> generally the case when you cannot eliminate SNAPSHOTs from your  
> build. When you see SNAPSHOT not from your project in our POM it is  
> a warning sign.

Dunno about that... the most tightly coupled SNAPSHOT we have now is  
OpenEJB... and really, I think that its so tightly coupled that it  
really should be in the same codebase.  BUT, that is not what other  
key plays believe, so then we are left trying to work the build  
system to generate something that is predictable given the  
circumstances.  Unfortunatly, mvn does not have a good answer for  
this problem... all it has is SNAPSHOT... and IMO that is not worth a  
damn.  Its more trouble than it is worth... and more so since all you  
people are using it everywhere, its very difficult to educate people  
to its harm.


> You guys seem to want to use the release plugin for releasing  
> things with SNAPSHOTs in them. That's not what it was made for and  
> again it's a function of you not managing your relationships with  
> your dependent projects. A release with SNAPSHOTs is not a release.  
> Tagging something that contains a POM with a SNAPSHOT identifier is  
> completely useless. I mean look at your 1.1 release:
>
> http://svn.apache.org/repos/asf/geronimo/server/tags/1.1.1/
>
> That's insane. That's a tag! How did you do a release with  
> SNAPSHOTs? That's not Maven's fault if you can't built that again  
> if you were careless enough to tag something with SNAPSHOTs. You  
> have got to use releases of plugins, parent POMs, and dependences  
> more frequently. Maven has lots of bugs, sure, but this mess is  
> largely your own doing. There are lots of Maven users with way more  
> source code and modules then you and don't have nearly the number  
> of problems you do.

I had nothing to do with any of that release... I have been trying to  
unfuck a bunch of issues with the release process in general.... and  
for the record its not all mvn related stuff that needs unfucking.   
Its more policy on how one uses mvn that needs work... some issues  
with the default plugins, some bugs, but more than anything it more  
the policy about how one uses mvn to build/release projects rather  
than mvn itself.


>> I really feel that the little short-cuts and side-steps around  
>> maven problems are going to kill us.  And really, I don't see how  
>> any medium to large sized group can effectively use maven to  
>> manage releases of their projects...
>
> Lots of projects do it fine. Your dependency chain is fragile which  
> is the problem. Your coupling is too high which will not let you do  
> releases as you have to, or you are not working with the producers  
> of your dependencies to create releases. In trunk right now you  
> have a SNAPSHOT for the POM that seems to contain the dependencies  
> for everything you need. That's wrong. You need to release those  
> parent POMs and the dependencies in them. It is natural to have  
> SNAPSHOTs for the project in question, for CI and while you're  
> developing. But a SNAPSHOT in a dependency means that you, as a  
> last resort, had to use something in flux because it cannot work  
> otherwise. This immediately should signal you to contact that  
> project to fix it, and release it. And again, because all your  
> dependent projects seem to share many of the developers you should  
> be able to do that.

Lots of projects do fine... sure.  Smaller projects with less  
requirement for durable/repeatable and stable builds should work  
fine.  But larger projects, with more moving parts that need to have  
builds repeatable over 10 years+ (with no changes to tags) will  
almost certainly fall flat on their faces when using mvn... UNLESS  
certain precautions and limitations are heeded.

The same is true for almost any other build tool... make and ant both  
can end up in the same situation... BUT... the key difference is that  
mvn makes it way to easy to get into a situation where your builds  
are non-durable and non-stable right out of the box.  Its got  
instabliltiy and non-durability built right into it with its remote  
repository... a problem which make and ant do not suffer from... and  
a reason why ant is more prevalent in commercial shops than mvn is.

  * * *

Anyways, I'm gonna stop here.  I could certainly go on and respond to  
everything in this mail... but I don't see the point.

My purpose was not to diss mvn here... but more so to suggest that  
there are more effective ways to use mvn which fit better into our  
integration model... there are better ways to use mvn ASIS with the  
requirements/restrictions we have now.

But even more so... my point was... that changes that have been made  
recently have really fucked over the work I have been doing to  
produce automated reliable durable builds for our projects... and  
yes, based on all those SNAPSHOTS of all those projects.

I had it working and it was working well... until the model of  
versioning specs changed... more specifically since the specs/trunk  
branch could not actually build and now it builds artifacts which are  
not being using by anything... the bits that are needed are now only  
available to build off of little spec module branches, which is a  
huge pain to automate.

IMO the only problem specific to maven here.... is that this  
versioning model was allowed by maven and even more so promoted by  
mvn since that is  how you folks manage your plugin modules.  So we  
have the lemmings following by example... right off the cliff.

And lastly... none of this is anything I consider mine... or my  
own... its all junk that I have picked up and been trying to fix.   
And at one point I did believe that mvn was going to help me do  
that... in retrospect... Ant would have been a better tool.... less  
bugs... less mystery... less time trying to figure out just what the  
fuck broke.

I cant' even count the number of times I have had to nuke my local  
repo, rebuld specs, openejb, g by hand to debug someone else's mvn  
problem... even more prevalent is the number of times I have heard of  
other people doing the same.  We spend more time fucking with mvn  
than we do actually writing code for Geronimo... and that is  
certainly a sign that something is wrong.

--jason

Mime
View raw message