ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mitch Gitman <mgit...@gmail.com>
Subject Re: Efficient dependency rebuilds for development and debugging
Date Tue, 19 Jun 2012 20:06:43 GMT
So many of these issues are taken care of with best-practice use of a
continuous integration server that publishes integration versions of the
various modules to a shared Ivy repository. The Ivy caching strategy for
integration versions should be to never trust the local cache and always
check for the latest. Likewise, if someone commits a change to project C,
your CI server should be configured to recursively build dependent jobs on
successful builds. And one more requirement: at least in the source
(unpublished) ivy.xml files, you need to express the versions of
integration-level dependencies as ranges using either the + syntax or an
upper bound.

For local developer builds, so long as you never trust the cache on
integration versions, you should be able to locally build and publish C,
then locally build and publish B with the latest C, then locally build,
test, and publish A with the latest B and C. If you don't bother to sync up
with C, though, then your Ivy settings should be configured to fall back to
the shared Ivy repository so you'll just pick up the latest version that
was built and published to the shared repo by the CI server.

You write: "For complicated projects which involve several direct and
indirect dependencies, this can be an error-prone process, since we need to
keep track of where a given change fits in the hierarchy and thus which
artifacts need to be rebuilt and which ivy.xml files need to be updated to
use those artifacts."

Really, there's nothing complicated or error-prone about this. Nor is there
any need to keep track of which Ivy modules need to be rebuilt. Everything
gets rebuilt cascading from least dependent upwards.

You write: "It also raises the problem of creating unique revision
identifiers for each updated module, which is painful to do manually and
dangerous to do automatically.  Specifically, using automatically-generated
revision IDs or mutable revisions can make the build non-reentrant; i.e.
two separate builds might accidentally use each other's revisions instead
of their own since they share the same repositories and possibly even the
same cache."

Integration versions published through a local dev build can always use
some hard-coded suffix like –dev or –SNAPSHOT, and you just overwrite every
time you publish. For CI builds to a shared Ivy repo, you do want to use an
automated suffix, and you do want to make it effectively unique, but
there's no real problem in doing so, if you do it right, when it comes to
making integration builds unique. The most common suffix I've seen used is
a timestamp. And each project's CI job should have its own dedicated Ivy
cache that gets wiped out every time. For CI builds, reliability trumps
performance.

On Tue, Jun 19, 2012 at 10:01 AM, Joel Dice <joel.dice@gmail.com> wrote:

> Hi all,
>
> We've recently adopted Ivy for dependency management after previously
> relying on source-based dependencies.  For example, we formerly used e.g.
> <ant dir="${basedir}/../some-**dependency/"/> to recursively build a
> dependency and then directly refered to the result of that build (e.g.
> ${basedir}/../some-dependency/**build/foo.jar).  Now we're using Ivy and
> Artifactory to manage our dependencies, but we've been struggling with the
> overhead and uncertainty that this process has introduced and are looking
> for advice.
>
> Previously, if we wanted to add a feature or debug a problem which might
> involve code from multiple projects, our debug cycle looked like this:
>
>  1. Change the code
>  2. Run ant in the top-level project
>  3. Test the result
>  4. Repeat
>
> The advantage of this process was that we could focus on the code itself
> and trust the build system to rebuild what we changed and anything that
> depended on it.  Now, however, it looks more like this:
>
>  1. Change the code
>  2. Publish a new artifact
>  3. Update the top-level project to use the new artifact
>  4. Test the result
>  5. Repeat
>
> Note that step 2 above might need to repeated several times for indirect
> dependencies, since if A depends on B, which depends on C, and C changes,
> new artifacts need to be built for both C and B.  For complicated projects
> which involve several direct and indirect dependencies, this can be an
> error-prone process, since we need to keep track of where a given change
> fits in the hierarchy and thus which artifacts need to be rebuilt and which
> ivy.xml files need to be updated to use those artifacts.
>
> It also raises the problem of creating unique revision identifiers for
> each updated module, which is painful to do manually and dangerous to do
> automatically.  Specifically, using automatically-generated revision IDs or
> mutable revisions can make the build non-reentrant; i.e. two separate
> builds might accidentally use each other's revisions instead of their own
> since they share the same repositories and possibly even the same cache.
> The result is that we don't have high confidence that the code we changed
> is the code we're testing, or that unrelated changes haven't slipped in
> accidentally.
>
> So, I'm looking for advice for how we can retain the advantages of using
> Ivy to manage our dependencies and also have the speed and simplicity of
> the recursive builds we used to use.
>
> Thanks.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message