maven-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Connolly <stephen.alan.conno...@gmail.com>
Subject Re: improving incremental builds
Date Mon, 27 Aug 2012 19:53:31 GMT
The issue (for benson) is when the shade plugin modifies after the class
files have changed... the jar plugin will currently skip recreating the
jar, and he has no way of knowing if the jar file has been shaded or
not.... the solution is to have the manifest.mf include an entry that
indicates created by shade, so that shade can safely skip

On 27 August 2012 19:39, Mark Struberg <struberg@yahoo.de> wrote:

> Not sure if I get the problem.
>
> The timestamp of the input files is still < than the timestamp of the jar
> file. Regardless if it later got modified by the shade-plugin or not.
>
> LieGrue,
> strub
>
>
>
>
> >________________________________
> > From: Stephen Connolly <stephen.alan.connolly@gmail.com>
> >To: Maven Developers List <dev@maven.apache.org>
> >Cc: Mark Struberg <struberg@yahoo.de>
> >Sent: Monday, August 27, 2012 8:11 PM
> >Subject: Re: improving incremental builds
> >
> >
> >On 27 August 2012 19:10, Stephen Connolly <
> stephen.alan.connolly@gmail.com> wrote:
> >
> >On 27 August 2012 18:09, Benson Margulies <bimargulies@gmail.com> wrote:
> >>
> >>On Mon, Aug 27, 2012 at 12:07 PM, Mark Struberg <struberg@yahoo.de>
> wrote:
> >>>> build1 and build 2 are different modules or different mvn invocations
> with some time inbetween?
> >>>>
> >>>> Of course we will need to test all plugins to behave incremental
> like! Means the maven-shade-plugin should check itself whether it needs
> shading or not at all.
> >>>
> >>>Two builds, separated in time, of exactly the same module.
> >>>
> >>>The problem here is that the jar plugin has already decided not to
> >>>rebuild the jar by the time the shade plugin gets there, and I have no
> >>>idea how to make the shade plugin play along.
> >>>
> >>
> >>
> >>Any chance we can get the jar plugin to inspect some of the artifacts
> that it generates into the jar to see if they are consistent with what the
> jar plugin produces...
> >>
> >>
> >>I'm looking at you META-INF/MANIFEST.MF
> >>
> >>
> >>I.O.W. if the
> >>
> >>
> >>Created-By: Apache Maven 3.0.4 (Maven JAR Plugin)
> >>
> >>
> >>is switched to
> >>
> >>
> >>Created-By: Apache Maven 3.0.4 (Maven Shade Plugin)
> >>
> >>
> >>Then the Maven JAR plugin can infer that the timestamp check is
> pointless, and force regen
> >>
> >
> >
> >And people customizing that header can just live with the issues that
> creates.
> >
> >-Stephen
> >>
> >>
> >>>
> >>>>
> >>>> But the worst thing happening would be that we compile too much.
> Thats still better than a full mvn clean install :)
> >>>>
> >>>> LieGrue,
> >>>> strub
> >>>>
> >>>>
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: Benson Margulies <bimargulies@gmail.com>
> >>>>> To: Maven Developers List <dev@maven.apache.org>; Mark Struberg
<
> struberg@yahoo.de>
> >>>>> Cc:
> >>>>> Sent: Monday, August 27, 2012 5:07 PM
> >>>>> Subject: Re: improving incremental builds
> >>>>>
> >>>>> On Mon, Aug 27, 2012 at 8:40 AM, Mark Struberg <struberg@yahoo.de>
> wrote:
> >>>>>>  We might implement this by storing the list of all activated
> profiles + all
> >>>>> resolved environment properties in a file in target/ somewhere.
> >>>>>>  That would be part of a.) in my previous mail to Romain.
> >>>>>
> >>>>> How about the situation with shade?
> >>>>>
> >>>>> Build 1 runs shade and it replaces the primary build artifact with
a
> shaded one.
> >>>>>
> >>>>> Build 2 sees that nothing has changed, but shade doesn't know, and
it
> >>>>> reshades, eating its own output with a ton of messages.
> >>>>>
> >>>>> Can I fix this in shade with current technologies?
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>  LieGrue,
> >>>>>>  strub
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>  ----- Original Message -----
> >>>>>>>  From: Anders Hammar <anders@hammar.net>
> >>>>>>>  To: Maven Developers List <dev@maven.apache.org>;
Mark Struberg
> >>>>> <struberg@yahoo.de>
> >>>>>>>  Cc:
> >>>>>>>  Sent: Monday, August 27, 2012 12:24 PM
> >>>>>>>  Subject: Re: improving incremental builds
> >>>>>>>
> >>>>>>>>   +0 for auto detecting changed scenarios. If someone
changes a
> >>>>> profile or
> >>>>>>>  the whole pom, then I'd expect he invokes a clean manually.
 We
> >>>>> have to
> >>>>>>>  document this expectation of course.
> >>>>>>>
> >>>>>>>  What do others think of this? I'm thinking it would be
awesome
> (but
> >>>>>>>  maybe difficult) if plugins could determine if its configuration
> has
> >>>>>>>  changed, and then handle this automatically. If there are
to many
> >>>>>>>  cases where a manual clean build is required, I believe
people
> will
> >>>>>>>  continue to do "mvn clean install" (which I see all the
time
> >>>>> with
> >>>>>>>  developers), which I think is what we try to get away from.
> >>>>>>>
> >>>>>>>  /Anders
> >>>>>>>
> >>>>>>>>
> >>>>>>>>   LieGrue,
> >>>>>>>>   strub
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>   ----- Original Message -----
> >>>>>>>>>   From: Anders Hammar <anders@hammar.net>
> >>>>>>>>>   To: Maven Developers List <dev@maven.apache.org>
> >>>>>>>>>   Cc:
> >>>>>>>>>   Sent: Monday, August 27, 2012 9:57 AM
> >>>>>>>>>   Subject: Re: improving incremental builds
> >>>>>>>>>
> >>>>>>>>>>    I already started tweaking the m-compiler-plugin
and
> >>>>> found quite
> >>>>>>>  scary
> >>>>>>>>>   issues. There is e.g. MCOMPILER-21 (open since
2005) which
> >>>>> prevents
> >>>>>>>  incremental
> >>>>>>>>>   build and would kick in in your scenario.
> >>>>>>>>>
> >>>>>>>>>   This is interesting. I've looked a bit at Mojo's
JAXB
> >>>>> plugin in
> >>>>>>>  the
> >>>>>>>>>   context of detecting stale generated files (i.e.
need to
> >>>>> re-generate)
> >>>>>>>>>   and it's similar to this. It would extremely nice
if there
> >>>>> would be
> >>>>>>>  a
> >>>>>>>>>   common "framework" for handling incremental builds.
> >>>>>>>>>   In addition to what's been mentioned in the jira
(I just
> >>>>> browsed it
> >>>>>>>>>   quickly), we have cases when includes/excludes
change, the
> >>>>> sourceDir
> >>>>>>>>>   changes, etc which should trigger cleanup and
re-compilation.
> >>>>>>>>>
> >>>>>>>>>   /Anders
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>    I talked about 2 scenarios
> >>>>>>>>>>
> >>>>>>>>>>    a.) all phases >= package, e.g. mvn verify,
mvn
> >>>>> install, ...
> >>>>>>>  Here we
> >>>>>>>>>   have an artifact on the disc and could test for
the md5 of
> >>>>> them, right?
> >>>>>>>>>>
> >>>>>>>>>>    b.) all phases < package. This is e.g.
mvn compile or
> >>>>> mvn test.
> >>>>>>>  In that
> >>>>>>>>>   case we don't produce a jar/war/ear/... artifact
but only
> >>>>> get the
> >>>>>>>  files on
> >>>>>>>>>   the disk linked in MavenProject#getProjectArtifacts()...
> >>>>> file(). This
> >>>>>>>  is the
> >>>>>>>>>   case where the maven-compiler-plugin kicks in.
I'm
> >>>>> currently in the
> >>>>>>>  process
> >>>>>>>>>   of rewriting big parts of it, mainly I introduced
> >>>>>>>>>>      b.1 a check for project.artifacts/project.testArtifacts
> >>>>> .file()
> >>>>>>>  is a
> >>>>>>>>>   local file path .isDirectory() and has files which
are newer
> >>>>> (actually
> >>>>>>>>  =)
> >>>>>>>>>   than the buildStarted timestamp.
> >>>>>>>>>>      b.2 check whether there are any *.java
(sourceincludes)
> >>>>> files
> >>>>>>>  which are
> >>>>>>>>>   newer than their class files.
> >>>>>>>>>>
> >>>>>>>>>>    In both cases I now force a FULL recompile
of the module!
> >>>>>>>  Compiling only
> >>>>>>>>>   parts produced classes which are actually broken!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>    My approach is the following: compiling a
bit too much is
> >>>>> better
> >>>>>>>  than
> >>>>>>>>>   missing some files which need compilation. Because
the later
> >>>>> is exactly
> >>>>>>>  the
> >>>>>>>>>   reason why noone uses mvn compile but always do
a mvn clean
> >>>>> compile
> >>>>>>>  nowadays. If
> >>>>>>>>>   mvn compile is reliable, then we will end up being
much faster
> >>>>> than an
> >>>>>>>>>   unconditional full compile on the whole project!
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>    LieGrue,
> >>>>>>>>>>    strub
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>    PS: We need to move all our plugins which
still use the
> >>>>>>>>>   plugin-testing-harness to the maven-invoker-plugin
as the
> >>>>> test-harness
> >>>>>>>  is broken
> >>>>>>>>>   with sisu. (sisu changed the 'container' field
from
> >>>>> plexus
> >>>>>>>  original:
> >>>>>>>>>   protected to 'private')
> >>>>>>>>>>
> >>>>>>>>>>    PPS: how do we maintain the plexus-compiler-utils
stuff?
> >>>>> This
> >>>>>>>  contains some
> >>>>>>>>>   weirdo bugs which should get fixed...
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>    ----- Original Message -----
> >>>>>>>>>>>    From: Olivier Lamy <olamy@apache.org>
> >>>>>>>>>>>    To: Maven Developers List
> >>>>> <dev@maven.apache.org>; Mark
> >>>>>>>  Struberg
> >>>>>>>>>   <struberg@yahoo.de>
> >>>>>>>>>>>    Cc:
> >>>>>>>>>>>    Sent: Monday, August 27, 2012 9:13 AM
> >>>>>>>>>>>    Subject: Re: improving incremental builds
> >>>>>>>>>>>
> >>>>>>>>>>>    Hi,
> >>>>>>>>>>>    Sounds good idea trying to improve that.
> >>>>>>>>>>>    My question: on what is based the md5
calculation ?
> >>>>>>>>>>>    If people don't use 'install' but only
> >>>>>>>  'test'
> >>>>>>>>>   (perso I use
> >>>>>>>>>>>    that to
> >>>>>>>>>>>    avoid too much io with jar creation etc..),
so in
> >>>>> such case we
> >>>>>>>  cannot
> >>>>>>>>>>>    do md5 calculation on the produced artifact.
> >>>>>>>>>>>
> >>>>>>>>>>>    2012/8/26 Mark Struberg <struberg@yahoo.de>:
> >>>>>>>>>>>>     Hi David!
> >>>>>>>>>>>>
> >>>>>>>>>>>>>     So your idea is to find the
list of changed
> >>>>> modules
> >>>>>>>  and
> >>>>>>>>>>>>>     then build that list with -amd?
> >>>>>>>>>>>>     Yes, kind of.
> >>>>>>>>>>>>
> >>>>>>>>>>>>     At the moment mvn -amd builds the
dependents of
> >>>>> the
> >>>>>>>  _current_
> >>>>>>>>>   module. If
> >>>>>>>>>>>    you use my example and change BeanA.java,
then run
> >>>>> mvn -amd
> >>>>>>>  from the
> >>>>>>>>>   root module
> >>>>>>>>>>>    you will see that only moduleA gets rebuild
and
> >>>>> moduleB
> >>>>>>>  remains
> >>>>>>>>>   original.
> >>>>>>>>>>>    Because moduleB is not a dependent of
the root
> >>>>> module.
> >>>>>>>>>>>>
> >>>>>>>>>>>>     But yes, I'm completely with you
that we
> >>>>> have most of
> >>>>>>>  the
> >>>>>>>>>   ingredients
> >>>>>>>>>>>    in the maven codebase already. At least
for the start
> >>>>> we could
> >>>>>>>  easily
> >>>>>>>>>   detect
> >>>>>>>>>>>    changed build results and add those to
the
> >>>>> 'amd' list.
> >>>>>>>  That
> >>>>>>>>>   would
> >>>>>>>>>>>    already be much better than what we have
today imo.
> >>>>> And this
> >>>>>>>  should be
> >>>>>>>>>   pretty
> >>>>>>>>>>>    easy to implement.
> >>>>>>>>>>>>
> >>>>>>>>>>>>     Indeed, what I proposed goes a bit
beyond -amd.
> >>>>> I would
> >>>>>>>  not only
> >>>>>>>>>   check the
> >>>>>>>>>>>    current build tree like -amd does (even
if that would
> >>>>> be a
> >>>>>>>  good start)
> >>>>>>>>>   but
> >>>>>>>>>>>    remember the md5 of all the dependencies
of each
> >>>>> module
> >>>>>>>  (simply store
> >>>>>>>>>   them in a
> >>>>>>>>>>>    file in ./target/) And if you find a
dependency which
> >>>>> is not
> >>>>>>>  in the
> >>>>>>>>>   list (e.g.
> >>>>>>>>>>>    after upgrade from one version to another)
or has a
> >>>>> different
> >>>>>>>  md5 (this
> >>>>>>>>>   covers
> >>>>>>>>>>>    SNAPSHOT versions) , then we could force
a full
> >>>>> rebuild (mvn
> >>>>>>>  -amd) of
> >>>>>>>>>   this
> >>>>>>>>>>>    module.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>     LieGrue,
> >>>>>>>>>>>>     strub
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>     ----- Original Message -----
> >>>>>>>>>>>>>     From: David Jencks
> >>>>> <david_jencks@yahoo.com>
> >>>>>>>>>>>>>     To: Maven Developers List
> >>>>>>>  <dev@maven.apache.org>
> >>>>>>>>>>>>>     Cc:
> >>>>>>>>>>>>>     Sent: Sunday, August 26, 2012
8:34 AM
> >>>>>>>>>>>>>     Subject: Re: improving incremental
builds
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>     Is what you want different from
what
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>     mvn -amd moduleA
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>     does?  So your idea is to find
the list of
> >>>>> changed
> >>>>>>>  modules and
> >>>>>>>>>   then
> >>>>>>>>>>>    build that
> >>>>>>>>>>>>>     list with -amd?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>     thanks
> >>>>>>>>>>>>>     david jencks
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>     On Aug 25, 2012, at 1:32 PM,
Mark Struberg
> >>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>      Hi folks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      After a short discussion
with Kristian,
> >>>>> I've
> >>>>>>>  created
> >>>>>>>>>   a small
> >>>>>>>>>>>    app with 2
> >>>>>>>>>>>>>     modules which shows a few problems
with
> >>>>> mavens
> >>>>>>>  incremental
> >>>>>>>>>   build logic.
> >>>>>>>>>>>>>>      And since incremental builds
do not
> >>>>> work well,
> >>>>>>>  people use
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      $> mvn _clean_ install
> >>>>>>>>>>>>>>      all the time.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      We could speed up the development
> >>>>> effort heavily
> >>>>>>>  if we
> >>>>>>>>>   make
> >>>>>>>>>>>>>>      $> mvn  install
> >>>>>>>>>>>>>>      (without an upfront clean)
more
> >>>>> reliable.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      The sample [1] consists
of moduleA and
> >>>>> moduleB
> >>>>>>>  with BeanA
> >>>>>>>>>   and
> >>>>>>>>>>>    BeanB
> >>>>>>>>>>>>>     respectively.
> >>>>>>>>>>>>>>      BeanB has a private BeanA
field and
> >>>>> moduleB has
> >>>>>>>  a
> >>>>>>>>>   dependency on
> >>>>>>>>>>>    moduleA.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      If I change BeanA and just
invoke mvn
> >>>>> install
> >>>>>>>  then only
> >>>>>>>>>   moduleA
> >>>>>>>>>>>    gets
> >>>>>>>>>>>>>     rebuilt. We currently do not
rebuild
> >>>>> moduleB, but we
> >>>>>>>  should do
> >>>>>>>>>   to
> >>>>>>>>>>>    create a
> >>>>>>>>>>>>>     reliable output.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      In fact, the incremental
build within a
> >>>>> single
> >>>>>>>  module
> >>>>>>>>>   already
> >>>>>>>>>>>    works to some
> >>>>>>>>>>>>>     degrees, but we must detect
a change in
> >>>>> dependencies
> >>>>>>>  and
> >>>>>>>>>   trigger a full
> >>>>>>>>>>>    rebuild
> >>>>>>>>>>>>>     on all depending modules. This
could be done
> >>>>> by
> >>>>>>>  storing the
> >>>>>>>>>   md5 of all
> >>>>>>>>>>>>>     dependency artifacts and compare
them on the
> >>>>> next
> >>>>>>>  build. If
> >>>>>>>>>   the md5 of
> >>>>>>>>>>>    a
> >>>>>>>>>>>>>     dependency did change, then
we need to build
> >>>>> the
> >>>>>>>  module full
> >>>>>>>>>   cycle.
> >>>>>>>>>>>>>>      Other ideas are welcome.
Slaps as well
> >>>>> if I
> >>>>>>>  forgot some
> >>>>>>>>>   obvious
> >>>>>>>>>>>    stuff and
> >>>>>>>>>>>>>     all works well already.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      LieGrue,
> >>>>>>>>>>>>>>      strub
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
>  ---------------------------------------------------------------------
> >>>>>>>>>>>>>>      To unsubscribe, e-mail:
> >>>>>>>  dev-unsubscribe@maven.apache.org
> >>>>>>>>>>>>>>      For additional commands,
e-mail:
> >>>>>>>>>   dev-help@maven.apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>     To unsubscribe, e-mail:
> >>>>>>>  dev-unsubscribe@maven.apache.org
> >>>>>>>>>>>>>     For additional commands, e-mail:
> >>>>>>>  dev-help@maven.apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>     To unsubscribe, e-mail:
> >>>>> dev-unsubscribe@maven.apache.org
> >>>>>>>>>>>>     For additional commands, e-mail:
> >>>>>>>  dev-help@maven.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>    --
> >>>>>>>>>>>    Olivier Lamy
> >>>>>>>>>>>    Talend: http://coders.talend.com
> >>>>>>>>>>>   http://twitter.com/olamy |
> >>>>> http://linkedin.com/in/olamy
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
>  ---------------------------------------------------------------------
> >>>>>>>>>>    To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>>>>>    For additional commands, e-mail:
> >>>>> dev-help@maven.apache.org
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>>   To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>>>>   For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>   To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>>>   For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>>>
> >>>>>>>
> >>>>>>>
>  ---------------------------------------------------------------------
> >>>>>>>  To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>>  For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
>  ---------------------------------------------------------------------
> >>>>>>  To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>>>  For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>> For additional commands, e-mail: dev-help@maven.apache.org
> >>>>
> >>>
> >>>---------------------------------------------------------------------
> >>>To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>>For additional commands, e-mail: dev-help@maven.apache.org
> >>>
> >>>
> >>
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message