ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Dice <joel.d...@gmail.com>
Subject Re: Efficient dependency rebuilds for development and debugging
Date Tue, 19 Jun 2012 22:48:01 GMT
Hi Mitch,

Thanks for your response.  Please see my comments below.

On Tue, 19 Jun 2012, Mitch Gitman wrote:

> So many of these issues are taken care of with best-practice use of a
> continuous integration server that publishes integration versions of the
> various modules to a shared Ivy repository. The Ivy caching strategy for
> integration versions should be to never trust the local cache and always
> check for the latest. Likewise, if someone commits a change to project C,
> your CI server should be configured to recursively build dependent jobs on
> successful builds. And one more requirement: at least in the source
> (unpublished) ivy.xml files, you need to express the versions of
> integration-level dependencies as ranges using either the + syntax or an
> upper bound.

Yes, that's exactly what we're doing on the CI server, and it works great. 
Where we're having trouble is the local developer builds, which are more 
frequent in practice since it may take dozens of iterations in a debug 
cycle to get the code to the point where it's ready to commit.

> For local developer builds, so long as you never trust the cache on
> integration versions, you should be able to locally build and publish C,
> then locally build and publish B with the latest C, then locally build,
> test, and publish A with the latest B and C. If you don't bother to sync up
> with C, though, then your Ivy settings should be configured to fall back to
> the shared Ivy repository so you'll just pick up the latest version that
> was built and published to the shared repo by the CI server.
>
> You write: "For complicated projects which involve several direct and
> indirect dependencies, this can be an error-prone process, since we need to
> keep track of where a given change fits in the hierarchy and thus which
> artifacts need to be rebuilt and which ivy.xml files need to be updated to
> use those artifacts."
>
> Really, there's nothing complicated or error-prone about this. Nor is there
> any need to keep track of which Ivy modules need to be rebuilt. Everything
> gets rebuilt cascading from least dependent upwards.

Yes, that's what happens on the CI server.  Are you suggesting that each 
developer run a CI server on his or her development machine and let it 
take care of cascading changes?

Also, I don't necessarily want to rebuild every module that depends 
directly or indirectly on the one I just changed.  Instead, I normally 
want to rebuild the minimum amount of code needed to test a specific 
top-level (i.e. most-dependent) project.  Once I'm happy with my change as 
far as it concerns that project, I can kick off builds of other projects 
that depend on the module(s) I've changed and run their respective tests, 
but it make take several iterations of debugging the initial project 
before I get to that point.

> You write: "It also raises the problem of creating unique revision
> identifiers for each updated module, which is painful to do manually and
> dangerous to do automatically.  Specifically, using automatically-generated
> revision IDs or mutable revisions can make the build non-reentrant; i.e.
> two separate builds might accidentally use each other's revisions instead
> of their own since they share the same repositories and possibly even the
> same cache."
>
> Integration versions published through a local dev build can always use
> some hard-coded suffix like =E2=80=93dev or =E2=80=93SNAPSHOT, and you just=
> overwrite every time you publish.

I don't think that will give me reentrant builds, though.  If I kick off 
two builds at around the same time (say the build takes a while and I want 
to build and test two separate changes in parallel), it's 
non-deterministic as to what I'll end up with when the most-dependent 
module is built.

> For CI builds to a shared Ivy repo, you do want to use an
> automated suffix, and you do want to make it effectively unique, but
> there's no real problem in doing so, if you do it right, when it comes to
> making integration builds unique. The most common suffix I've seen used is
> a timestamp. And each project's CI job should have its own dedicated Ivy
> cache that gets wiped out every time. For CI builds, reliability trumps
> performance.
>
>
> On Tue, Jun 19, 2012 at 10:01 AM, Joel Dice <joel.dice@gmail.com> wrote:
>
>> Hi all,
>>
>> We've recently adopted Ivy for dependency management after previously
>> relying on source-based dependencies.  For example, we formerly used e.g.
>> <ant dir=3D"${basedir}/../some-**dependency/"/> to recursively build a
>> dependency and then directly refered to the result of that build (e.g.
>> ${basedir}/../some-dependency/**build/foo.jar).  Now we're using Ivy and
>> Artifactory to manage our dependencies, but we've been struggling with th=
> e
>> overhead and uncertainty that this process has introduced and are looking
>> for advice.
>>
>> Previously, if we wanted to add a feature or debug a problem which might
>> involve code from multiple projects, our debug cycle looked like this:
>>
>>  1. Change the code
>>  2. Run ant in the top-level project
>>  3. Test the result
>>  4. Repeat
>>
>> The advantage of this process was that we could focus on the code itself
>> and trust the build system to rebuild what we changed and anything that
>> depended on it.  Now, however, it looks more like this:
>>
>>  1. Change the code
>>  2. Publish a new artifact
>>  3. Update the top-level project to use the new artifact
>>  4. Test the result
>>  5. Repeat
>>
>> Note that step 2 above might need to repeated several times for indirect
>> dependencies, since if A depends on B, which depends on C, and C changes,
>> new artifacts need to be built for both C and B.  For complicated project=
> s
>> which involve several direct and indirect dependencies, this can be an
>> error-prone process, since we need to keep track of where a given change
>> fits in the hierarchy and thus which artifacts need to be rebuilt and whi=
> ch
>> ivy.xml files need to be updated to use those artifacts.
>>
>> It also raises the problem of creating unique revision identifiers for
>> each updated module, which is painful to do manually and dangerous to do
>> automatically.  Specifically, using automatically-generated revision IDs =
> or
>> mutable revisions can make the build non-reentrant; i.e. two separate
>> builds might accidentally use each other's revisions instead of their own
>> since they share the same repositories and possibly even the same cache.
>> The result is that we don't have high confidence that the code we changed
>> is the code we're testing, or that unrelated changes haven't slipped in
>> accidentally.
>>
>> So, I'm looking for advice for how we can retain the advantages of using
>> Ivy to manage our dependencies and also have the speed and simplicity of
>> the recursive builds we used to use.
>>
>> Thanks.
>>
>
> --14dae9cfc83049b7fd04c2d8d219--
>
>

Mime
View raw message