maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Struberg <strub...@yahoo.de>
Subject Re: improving incremental builds
Date Mon, 27 Aug 2012 18:39:36 GMT
Not sure if I get the problem.

The timestamp of the input files is still < than the timestamp of the jar file. Regardless
if it later got modified by the shade-plugin or not.

LieGrue,
strub




>________________________________
> From: Stephen Connolly <stephen.alan.connolly@gmail.com>
>To: Maven Developers List <dev@maven.apache.org> 
>Cc: Mark Struberg <struberg@yahoo.de> 
>Sent: Monday, August 27, 2012 8:11 PM
>Subject: Re: improving incremental builds
> 
>
>On 27 August 2012 19:10, Stephen Connolly <stephen.alan.connolly@gmail.com> wrote:
>
>On 27 August 2012 18:09, Benson Margulies <bimargulies@gmail.com> wrote:
>>
>>On Mon, Aug 27, 2012 at 12:07 PM, Mark Struberg <struberg@yahoo.de> wrote:
>>>> build1 and build 2 are different modules or different mvn invocations with
some time inbetween?
>>>>
>>>> Of course we will need to test all plugins to behave incremental like! Means
the maven-shade-plugin should check itself whether it needs shading or not at all.
>>>
>>>Two builds, separated in time, of exactly the same module.
>>>
>>>The problem here is that the jar plugin has already decided not to
>>>rebuild the jar by the time the shade plugin gets there, and I have no
>>>idea how to make the shade plugin play along.
>>>
>>
>>
>>Any chance we can get the jar plugin to inspect some of the artifacts that it generates
into the jar to see if they are consistent with what the jar plugin produces...
>>
>>
>>I'm looking at you META-INF/MANIFEST.MF
>>
>>
>>I.O.W. if the 
>>
>>
>>Created-By: Apache Maven 3.0.4 (Maven JAR Plugin)
>>
>>
>>is switched to 
>>
>>
>>Created-By: Apache Maven 3.0.4 (Maven Shade Plugin)
>>
>>
>>Then the Maven JAR plugin can infer that the timestamp check is pointless, and force
regen
>>
>
>
>And people customizing that header can just live with the issues that creates.
> 
>-Stephen
>> 
>>
>>>
>>>>
>>>> But the worst thing happening would be that we compile too much. Thats still
better than a full mvn clean install :)
>>>>
>>>> LieGrue,
>>>> strub
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: Benson Margulies <bimargulies@gmail.com>
>>>>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg
<struberg@yahoo.de>
>>>>> Cc:
>>>>> Sent: Monday, August 27, 2012 5:07 PM
>>>>> Subject: Re: improving incremental builds
>>>>>
>>>>> On Mon, Aug 27, 2012 at 8:40 AM, Mark Struberg <struberg@yahoo.de>
wrote:
>>>>>>  We might implement this by storing the list of all activated profiles
+ all
>>>>> resolved environment properties in a file in target/ somewhere.
>>>>>>  That would be part of a.) in my previous mail to Romain.
>>>>>
>>>>> How about the situation with shade?
>>>>>
>>>>> Build 1 runs shade and it replaces the primary build artifact with a
shaded one.
>>>>>
>>>>> Build 2 sees that nothing has changed, but shade doesn't know, and it
>>>>> reshades, eating its own output with a ton of messages.
>>>>>
>>>>> Can I fix this in shade with current technologies?
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>  LieGrue,
>>>>>>  strub
>>>>>>
>>>>>>
>>>>>>
>>>>>>  ----- Original Message -----
>>>>>>>  From: Anders Hammar <anders@hammar.net>
>>>>>>>  To: Maven Developers List <dev@maven.apache.org>; Mark
Struberg
>>>>> <struberg@yahoo.de>
>>>>>>>  Cc:
>>>>>>>  Sent: Monday, August 27, 2012 12:24 PM
>>>>>>>  Subject: Re: improving incremental builds
>>>>>>>
>>>>>>>>   +0 for auto detecting changed scenarios. If someone changes
a
>>>>> profile or
>>>>>>>  the whole pom, then I'd expect he invokes a clean manually.
 We
>>>>> have to
>>>>>>>  document this expectation of course.
>>>>>>>
>>>>>>>  What do others think of this? I'm thinking it would be awesome
(but
>>>>>>>  maybe difficult) if plugins could determine if its configuration
has
>>>>>>>  changed, and then handle this automatically. If there are to
many
>>>>>>>  cases where a manual clean build is required, I believe people
will
>>>>>>>  continue to do "mvn clean install" (which I see all the time
>>>>> with
>>>>>>>  developers), which I think is what we try to get away from.
>>>>>>>
>>>>>>>  /Anders
>>>>>>>
>>>>>>>>
>>>>>>>>   LieGrue,
>>>>>>>>   strub
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>   ----- Original Message -----
>>>>>>>>>   From: Anders Hammar <anders@hammar.net>
>>>>>>>>>   To: Maven Developers List <dev@maven.apache.org>
>>>>>>>>>   Cc:
>>>>>>>>>   Sent: Monday, August 27, 2012 9:57 AM
>>>>>>>>>   Subject: Re: improving incremental builds
>>>>>>>>>
>>>>>>>>>>    I already started tweaking the m-compiler-plugin
and
>>>>> found quite
>>>>>>>  scary
>>>>>>>>>   issues. There is e.g. MCOMPILER-21 (open since 2005)
which
>>>>> prevents
>>>>>>>  incremental
>>>>>>>>>   build and would kick in in your scenario.
>>>>>>>>>
>>>>>>>>>   This is interesting. I've looked a bit at Mojo's JAXB
>>>>> plugin in
>>>>>>>  the
>>>>>>>>>   context of detecting stale generated files (i.e. need
to
>>>>> re-generate)
>>>>>>>>>   and it's similar to this. It would extremely nice
if there
>>>>> would be
>>>>>>>  a
>>>>>>>>>   common "framework" for handling incremental builds.
>>>>>>>>>   In addition to what's been mentioned in the jira (I
just
>>>>> browsed it
>>>>>>>>>   quickly), we have cases when includes/excludes change,
the
>>>>> sourceDir
>>>>>>>>>   changes, etc which should trigger cleanup and re-compilation.
>>>>>>>>>
>>>>>>>>>   /Anders
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    I talked about 2 scenarios
>>>>>>>>>>
>>>>>>>>>>    a.) all phases >= package, e.g. mvn verify,
mvn
>>>>> install, ...
>>>>>>>  Here we
>>>>>>>>>   have an artifact on the disc and could test for the
md5 of
>>>>> them, right?
>>>>>>>>>>
>>>>>>>>>>    b.) all phases < package. This is e.g. mvn
compile or
>>>>> mvn test.
>>>>>>>  In that
>>>>>>>>>   case we don't produce a jar/war/ear/... artifact but
only
>>>>> get the
>>>>>>>  files on
>>>>>>>>>   the disk linked in MavenProject#getProjectArtifacts()...
>>>>> file(). This
>>>>>>>  is the
>>>>>>>>>   case where the maven-compiler-plugin kicks in. I'm
>>>>> currently in the
>>>>>>>  process
>>>>>>>>>   of rewriting big parts of it, mainly I introduced
>>>>>>>>>>      b.1 a check for project.artifacts/project.testArtifacts
>>>>> .file()
>>>>>>>  is a
>>>>>>>>>   local file path .isDirectory() and has files which
are newer
>>>>> (actually
>>>>>>>>  =)
>>>>>>>>>   than the buildStarted timestamp.
>>>>>>>>>>      b.2 check whether there are any *.java (sourceincludes)
>>>>> files
>>>>>>>  which are
>>>>>>>>>   newer than their class files.
>>>>>>>>>>
>>>>>>>>>>    In both cases I now force a FULL recompile of
the module!
>>>>>>>  Compiling only
>>>>>>>>>   parts produced classes which are actually broken!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    My approach is the following: compiling a bit
too much is
>>>>> better
>>>>>>>  than
>>>>>>>>>   missing some files which need compilation. Because
the later
>>>>> is exactly
>>>>>>>  the
>>>>>>>>>   reason why noone uses mvn compile but always do a
mvn clean
>>>>> compile
>>>>>>>  nowadays. If
>>>>>>>>>   mvn compile is reliable, then we will end up being
much faster
>>>>> than an
>>>>>>>>>   unconditional full compile on the whole project!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    LieGrue,
>>>>>>>>>>    strub
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    PS: We need to move all our plugins which still
use the
>>>>>>>>>   plugin-testing-harness to the maven-invoker-plugin
as the
>>>>> test-harness
>>>>>>>  is broken
>>>>>>>>>   with sisu. (sisu changed the 'container' field from
>>>>> plexus
>>>>>>>  original:
>>>>>>>>>   protected to 'private')
>>>>>>>>>>
>>>>>>>>>>    PPS: how do we maintain the plexus-compiler-utils
stuff?
>>>>> This
>>>>>>>  contains some
>>>>>>>>>   weirdo bugs which should get fixed...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>    ----- Original Message -----
>>>>>>>>>>>    From: Olivier Lamy <olamy@apache.org>
>>>>>>>>>>>    To: Maven Developers List
>>>>> <dev@maven.apache.org>; Mark
>>>>>>>  Struberg
>>>>>>>>>   <struberg@yahoo.de>
>>>>>>>>>>>    Cc:
>>>>>>>>>>>    Sent: Monday, August 27, 2012 9:13 AM
>>>>>>>>>>>    Subject: Re: improving incremental builds
>>>>>>>>>>>
>>>>>>>>>>>    Hi,
>>>>>>>>>>>    Sounds good idea trying to improve that.
>>>>>>>>>>>    My question: on what is based the md5 calculation
?
>>>>>>>>>>>    If people don't use 'install' but only
>>>>>>>  'test'
>>>>>>>>>   (perso I use
>>>>>>>>>>>    that to
>>>>>>>>>>>    avoid too much io with jar creation etc..),
so in
>>>>> such case we
>>>>>>>  cannot
>>>>>>>>>>>    do md5 calculation on the produced artifact.
>>>>>>>>>>>
>>>>>>>>>>>    2012/8/26 Mark Struberg <struberg@yahoo.de>:
>>>>>>>>>>>>     Hi David!
>>>>>>>>>>>>
>>>>>>>>>>>>>     So your idea is to find the list
of changed
>>>>> modules
>>>>>>>  and
>>>>>>>>>>>>>     then build that list with -amd?
>>>>>>>>>>>>     Yes, kind of.
>>>>>>>>>>>>
>>>>>>>>>>>>     At the moment mvn -amd builds the dependents
of
>>>>> the
>>>>>>>  _current_
>>>>>>>>>   module. If
>>>>>>>>>>>    you use my example and change BeanA.java,
then run
>>>>> mvn -amd
>>>>>>>  from the
>>>>>>>>>   root module
>>>>>>>>>>>    you will see that only moduleA gets rebuild
and
>>>>> moduleB
>>>>>>>  remains
>>>>>>>>>   original.
>>>>>>>>>>>    Because moduleB is not a dependent of the
root
>>>>> module.
>>>>>>>>>>>>
>>>>>>>>>>>>     But yes, I'm completely with you that
we
>>>>> have most of
>>>>>>>  the
>>>>>>>>>   ingredients
>>>>>>>>>>>    in the maven codebase already. At least
for the start
>>>>> we could
>>>>>>>  easily
>>>>>>>>>   detect
>>>>>>>>>>>    changed build results and add those to the
>>>>> 'amd' list.
>>>>>>>  That
>>>>>>>>>   would
>>>>>>>>>>>    already be much better than what we have
today imo.
>>>>> And this
>>>>>>>  should be
>>>>>>>>>   pretty
>>>>>>>>>>>    easy to implement.
>>>>>>>>>>>>
>>>>>>>>>>>>     Indeed, what I proposed goes a bit
beyond -amd.
>>>>> I would
>>>>>>>  not only
>>>>>>>>>   check the
>>>>>>>>>>>    current build tree like -amd does (even
if that would
>>>>> be a
>>>>>>>  good start)
>>>>>>>>>   but
>>>>>>>>>>>    remember the md5 of all the dependencies
of each
>>>>> module
>>>>>>>  (simply store
>>>>>>>>>   them in a
>>>>>>>>>>>    file in ./target/) And if you find a dependency
which
>>>>> is not
>>>>>>>  in the
>>>>>>>>>   list (e.g.
>>>>>>>>>>>    after upgrade from one version to another)
or has a
>>>>> different
>>>>>>>  md5 (this
>>>>>>>>>   covers
>>>>>>>>>>>    SNAPSHOT versions) , then we could force
a full
>>>>> rebuild (mvn
>>>>>>>  -amd) of
>>>>>>>>>   this
>>>>>>>>>>>    module.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>     LieGrue,
>>>>>>>>>>>>     strub
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>     ----- Original Message -----
>>>>>>>>>>>>>     From: David Jencks
>>>>> <david_jencks@yahoo.com>
>>>>>>>>>>>>>     To: Maven Developers List
>>>>>>>  <dev@maven.apache.org>
>>>>>>>>>>>>>     Cc:
>>>>>>>>>>>>>     Sent: Sunday, August 26, 2012 8:34
AM
>>>>>>>>>>>>>     Subject: Re: improving incremental
builds
>>>>>>>>>>>>>
>>>>>>>>>>>>>     Is what you want different from
what
>>>>>>>>>>>>>
>>>>>>>>>>>>>     mvn -amd moduleA
>>>>>>>>>>>>>
>>>>>>>>>>>>>     does?  So your idea is to find
the list of
>>>>> changed
>>>>>>>  modules and
>>>>>>>>>   then
>>>>>>>>>>>    build that
>>>>>>>>>>>>>     list with -amd?
>>>>>>>>>>>>>
>>>>>>>>>>>>>     thanks
>>>>>>>>>>>>>     david jencks
>>>>>>>>>>>>>
>>>>>>>>>>>>>     On Aug 25, 2012, at 1:32 PM, Mark
Struberg
>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>      Hi folks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      After a short discussion
with Kristian,
>>>>> I've
>>>>>>>  created
>>>>>>>>>   a small
>>>>>>>>>>>    app with 2
>>>>>>>>>>>>>     modules which shows a few problems
with
>>>>> mavens
>>>>>>>  incremental
>>>>>>>>>   build logic.
>>>>>>>>>>>>>>      And since incremental builds
do not
>>>>> work well,
>>>>>>>  people use
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      $> mvn _clean_ install
>>>>>>>>>>>>>>      all the time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      We could speed up the development
>>>>> effort heavily
>>>>>>>  if we
>>>>>>>>>   make
>>>>>>>>>>>>>>      $> mvn  install
>>>>>>>>>>>>>>      (without an upfront clean)
more
>>>>> reliable.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      The sample [1] consists of
moduleA and
>>>>> moduleB
>>>>>>>  with BeanA
>>>>>>>>>   and
>>>>>>>>>>>    BeanB
>>>>>>>>>>>>>     respectively.
>>>>>>>>>>>>>>      BeanB has a private BeanA
field and
>>>>> moduleB has
>>>>>>>  a
>>>>>>>>>   dependency on
>>>>>>>>>>>    moduleA.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      If I change BeanA and just
invoke mvn
>>>>> install
>>>>>>>  then only
>>>>>>>>>   moduleA
>>>>>>>>>>>    gets
>>>>>>>>>>>>>     rebuilt. We currently do not rebuild
>>>>> moduleB, but we
>>>>>>>  should do
>>>>>>>>>   to
>>>>>>>>>>>    create a
>>>>>>>>>>>>>     reliable output.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      In fact, the incremental
build within a
>>>>> single
>>>>>>>  module
>>>>>>>>>   already
>>>>>>>>>>>    works to some
>>>>>>>>>>>>>     degrees, but we must detect a change
in
>>>>> dependencies
>>>>>>>  and
>>>>>>>>>   trigger a full
>>>>>>>>>>>    rebuild
>>>>>>>>>>>>>     on all depending modules. This
could be done
>>>>> by
>>>>>>>  storing the
>>>>>>>>>   md5 of all
>>>>>>>>>>>>>     dependency artifacts and compare
them on the
>>>>> next
>>>>>>>  build. If
>>>>>>>>>   the md5 of
>>>>>>>>>>>    a
>>>>>>>>>>>>>     dependency did change, then we
need to build
>>>>> the
>>>>>>>  module full
>>>>>>>>>   cycle.
>>>>>>>>>>>>>>      Other ideas are welcome.
Slaps as well
>>>>> if I
>>>>>>>  forgot some
>>>>>>>>>   obvious
>>>>>>>>>>>    stuff and
>>>>>>>>>>>>>     all works well already.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>      LieGrue,
>>>>>>>>>>>>>>      strub
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>  ---------------------------------------------------------------------
>>>>>>>>>>>>>>      To unsubscribe, e-mail:
>>>>>>>  dev-unsubscribe@maven.apache.org
>>>>>>>>>>>>>>      For additional commands,
e-mail:
>>>>>>>>>   dev-help@maven.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>     To unsubscribe, e-mail:
>>>>>>>  dev-unsubscribe@maven.apache.org
>>>>>>>>>>>>>     For additional commands, e-mail:
>>>>>>>  dev-help@maven.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>     To unsubscribe, e-mail:
>>>>> dev-unsubscribe@maven.apache.org
>>>>>>>>>>>>     For additional commands, e-mail:
>>>>>>>  dev-help@maven.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>    --
>>>>>>>>>>>    Olivier Lamy
>>>>>>>>>>>    Talend: http://coders.talend.com
>>>>>>>>>>>   http://twitter.com/olamy |
>>>>> http://linkedin.com/in/olamy
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>  ---------------------------------------------------------------------
>>>>>>>>>>    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>>>>    For additional commands, e-mail:
>>>>> dev-help@maven.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>>>>>   To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>>>   For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>>>>>   To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>>   For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>  ---------------------------------------------------------------------
>>>>>>>  To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>>  For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>
>>>>>>
>>>>>>  ---------------------------------------------------------------------
>>>>>>  To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>  For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>
>>>---------------------------------------------------------------------
>>>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>>
>>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org


Mime
View raw message