maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anders Hammar <and...@hammar.net>
Subject Re: improving incremental builds
Date Mon, 27 Aug 2012 07:57:45 GMT
> I already started tweaking the m-compiler-plugin and found quite scary issues. There is
e.g. MCOMPILER-21 (open since 2005) which prevents incremental build and would kick in in
your scenario.

This is interesting. I've looked a bit at Mojo's JAXB plugin in the
context of detecting stale generated files (i.e. need to re-generate)
and it's similar to this. It would extremely nice if there would be a
common "framework" for handling incremental builds.
In addition to what's been mentioned in the jira (I just browsed it
quickly), we have cases when includes/excludes change, the sourceDir
changes, etc which should trigger cleanup and re-compilation.

/Anders

>
> I talked about 2 scenarios
>
> a.) all phases >= package, e.g. mvn verify, mvn install, ... Here we have an artifact
on the disc and could test for the md5 of them, right?
>
> b.) all phases < package. This is e.g. mvn compile or mvn test. In that case we don't
produce a jar/war/ear/... artifact but only get the files on the disk linked in MavenProject#getProjectArtifacts()...
file(). This is the case where the maven-compiler-plugin kicks in. I'm currently in the process
of rewriting big parts of it, mainly I introduced
>   b.1 a check for project.artifacts/project.testArtifacts .file() is a local file path
.isDirectory() and has files which are newer (actually >=) than the buildStarted timestamp.
>   b.2 check whether there are any *.java (sourceincludes) files which are newer than
their class files.
>
> In both cases I now force a FULL recompile of the module! Compiling only parts produced
classes which are actually broken!
>
>
> My approach is the following: compiling a bit too much is better than missing some files
which need compilation. Because the later is exactly the reason why noone uses mvn compile
but always do a mvn clean compile nowadays. If mvn compile is reliable, then we will end up
being much faster than an unconditional full compile on the whole project!
>
>
> LieGrue,
> strub
>
>
> PS: We need to move all our plugins which still use the plugin-testing-harness to the
maven-invoker-plugin as the test-harness is broken with sisu. (sisu changed the 'container'
field from plexus original: protected to 'private')
>
> PPS: how do we maintain the plexus-compiler-utils stuff? This contains some weirdo bugs
which should get fixed...
>
>
>
> ----- Original Message -----
>> From: Olivier Lamy <olamy@apache.org>
>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg <struberg@yahoo.de>
>> Cc:
>> Sent: Monday, August 27, 2012 9:13 AM
>> Subject: Re: improving incremental builds
>>
>> Hi,
>> Sounds good idea trying to improve that.
>> My question: on what is based the md5 calculation ?
>> If people don't use 'install' but only 'test' (perso I use
>> that to
>> avoid too much io with jar creation etc..), so in such case we cannot
>> do md5 calculation on the produced artifact.
>>
>> 2012/8/26 Mark Struberg <struberg@yahoo.de>:
>>>  Hi David!
>>>
>>>>  So your idea is to find the list of changed modules and
>>>>  then build that list with -amd?
>>>  Yes, kind of.
>>>
>>>  At the moment mvn -amd builds the dependents of the _current_ module. If
>> you use my example and change BeanA.java, then run mvn -amd from the root module
>> you will see that only moduleA gets rebuild and moduleB remains original.
>> Because moduleB is not a dependent of the root module.
>>>
>>>  But yes, I'm completely with you that we have most of the ingredients
>> in the maven codebase already. At least for the start we could easily detect
>> changed build results and add those to the 'amd' list. That would
>> already be much better than what we have today imo. And this should be pretty
>> easy to implement.
>>>
>>>  Indeed, what I proposed goes a bit beyond -amd. I would not only check the
>> current build tree like -amd does (even if that would be a good start) but
>> remember the md5 of all the dependencies of each module (simply store them in a
>> file in ./target/) And if you find a dependency which is not in the list (e.g.
>> after upgrade from one version to another) or has a different md5 (this covers
>> SNAPSHOT versions) , then we could force a full rebuild (mvn -amd) of this
>> module.
>>>
>>>
>>>  LieGrue,
>>>  strub
>>>
>>>
>>>
>>>  ----- Original Message -----
>>>>  From: David Jencks <david_jencks@yahoo.com>
>>>>  To: Maven Developers List <dev@maven.apache.org>
>>>>  Cc:
>>>>  Sent: Sunday, August 26, 2012 8:34 AM
>>>>  Subject: Re: improving incremental builds
>>>>
>>>>  Is what you want different from what
>>>>
>>>>  mvn -amd moduleA
>>>>
>>>>  does?  So your idea is to find the list of changed modules and then
>> build that
>>>>  list with -amd?
>>>>
>>>>  thanks
>>>>  david jencks
>>>>
>>>>  On Aug 25, 2012, at 1:32 PM, Mark Struberg wrote:
>>>>
>>>>>   Hi folks!
>>>>>
>>>>>   After a short discussion with Kristian, I've created a small
>> app with 2
>>>>  modules which shows a few problems with mavens incremental build logic.
>>>>>   And since incremental builds do not work well, people use
>>>>>
>>>>>   $> mvn _clean_ install
>>>>>   all the time.
>>>>>
>>>>>   We could speed up the development effort heavily if we make
>>>>>   $> mvn  install
>>>>>   (without an upfront clean) more reliable.
>>>>>
>>>>>
>>>>>   The sample [1] consists of moduleA and moduleB with BeanA and
>> BeanB
>>>>  respectively.
>>>>>   BeanB has a private BeanA field and moduleB has a dependency on
>> moduleA.
>>>>>
>>>>>   If I change BeanA and just invoke mvn install then only moduleA
>> gets
>>>>  rebuilt. We currently do not rebuild moduleB, but we should do to
>> create a
>>>>  reliable output.
>>>>>
>>>>>   In fact, the incremental build within a single module already
>> works to some
>>>>  degrees, but we must detect a change in dependencies and trigger a full
>> rebuild
>>>>  on all depending modules. This could be done by storing the md5 of all
>>>>  dependency artifacts and compare them on the next build. If the md5 of
>> a
>>>>  dependency did change, then we need to build the module full cycle.
>>>>>   Other ideas are welcome. Slaps as well if I forgot some obvious
>> stuff and
>>>>  all works well already.
>>>>>
>>>>>
>>>>>   LieGrue,
>>>>>   strub
>>>>>
>>>>>
>>>>>
>> ---------------------------------------------------------------------
>>>>>   To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>   For additional commands, e-mail: dev-help@maven.apache.org
>>>>>
>>>>
>>>>
>>>>  ---------------------------------------------------------------------
>>>>  To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>  For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>
>>>  ---------------------------------------------------------------------
>>>  To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>  For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>
>>
>>
>> --
>> Olivier Lamy
>> Talend: http://coders.talend.com
>> http://twitter.com/olamy | http://linkedin.com/in/olamy
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message