ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Suereth <>
Subject RE: cache busting and integration question
Date Mon, 23 Mar 2015 21:16:23 GMT
We experienced this bug in sbt as well.

If I recall correctly, it turns out that the implementation of most
resolvers is rather tied to the implementation of the cache.  One artifact
of this is that the chain resolver wasn't ever looking for -SNAPSHOTs when
we wanted it to.

We actually did the following, recently ->

1. A new chain resolver implementation which forces a date check against
the remote repository.   It turns out this exposed issues in publication
date not being a universal thing.  E.g. pom.xml files weren't reporting
legitimate publication dates, leading to any attempt to pull "latest" for a
snapshot being wrong.

2.  We created a new implementation of DependencyResolver which actually
uses aether.   This had two side effects:

2a. We avoid parent pom issues which still sneak in when using -SNAPSHOT
parents or weird profiles in POM's (why hadoop, why!!!)

2b. We always correctly update maven-metadata.xml and
maven-metadata-local.xml.  this means we'll have a somewhat legitimate
publication date to do loose comparisons on (not exact but good enough).

Both are still in experimental stage, but both now pass our previously
failing test cases around -SNAPSHOT updates, when used together.

What does this mean for ivy?

1. Ivy should probably support maven-metadata(-local).xml directly in its
maven code.
2. Ivy chain resolver / ivy cache may need tweaks for dynamic revisions to
make sure, even if the version number is the same, that remote repositories
(or just metadata.xml) is checked.

This adventure lead me to believe that the ivy cache api itself may need a
3.0 overhaul, and the maven code may need to be further separated from
"core" usage.  While the loss of reuse would be sad, I think the reality is
maven just doesn't work the way ivy was designed.
On Mar 23, 2015 12:33 PM, "Loren Kratzke" <> wrote:

> I am not getting the results that you describe. I am testing with a simple
> project that publishes a text file with the current date and time so it is
> very easy to see when I get a fresh or stale artifact. When I run using the
> configuration below, I get the following results when I publish my
> dependency and then consume it in the downstream project:
> [pub nexus] [get nexus] good
> [pub nexus] [get nexus] good
> [pub local] [get cached nexus artifact] bad
> [pub nexus] [get cached nexus artfiact] bad
> So as the first two tests indicate, I can get the latest snapshot from
> Nexus. That will work well for the integration server if we leave the local
> repo out.
> But as the 3rd and 4th tests indicate, if  I do a local publish then
> everything goes to hell. I do not get my local artifact, I get the cached
> artifact. And if I publish to Nexus again, I still get the cached artifact.
> In fact I get the cached artifact forever after a local publish no matter
> what gets published or where.
> That can't be desired behavior. There has to be a bug here somewhere, but
> I would be most happy with a work around. This behavior was witnessed when
> I debugged. It would find a local artifact, check its cache to find an ivy
> file that points at Nexus, and ultimately return a stale Nexus artifact
> from the cache.
> My setup is very simple - One Nexus repository and one local repository.
> Nothing fancy here. I have tried every combination of "checkmodified",
> "changingPattern", and "force" that the configuration can tolerate without
> it quacking a fur ball back at me but I cannot seem to get the desired
> behavior. Please find the err in my ways and show me where I have wandered
> off the ranch.
> I thought I had this working - once - and I changed something and can't
> seem to get it back. I thought I had it working as you described where it
> would pick up the snapshots, then it would pick up my local artifact, and
> it would always deliver the local artifact if present until the local cache
> was torched. That behavior is ok with me if I could get it back.
> Here is my settings.xml
> <ivysettings>
>     <caches defaultCacheDir="${user.home}/.ivy2/cache"/>
>     <settings defaultResolver="chain"/>
>     <credentials host="" realm="Sonatype Nexus
> Repository Manager" username="name" passwd="pass"/>
>     <property name="m2pattern"
> value="[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"/>
>     <property name="internal.repo.dir" value="${user.home}/.ivy2/m2"/>
>     <property name="mycompany.repo.url" value="
>     <resolvers>
>         <filesystem name="internal" force="true" checkmodified="true"
> changingPattern=".*">
>             <ivy
> pattern="${internal.repo.dir}/[module]/ivy-[revision].xml" />
>             <artifact pattern="${internal.repo.dir}/${m2pattern}" />
>         </filesystem>
>         <url name="nexus-snapshots-noplatform" m2compatible="true"
> checkmodified="true" changingPattern=".*">
>             <artifact
> pattern="${mycompany.repo.url}/snapshots/${m2pattern}" />
>         </url>
>         <chain name="chain" checkmodified="true">
>             <resolver ref="internal"/>
>             <resolver ref="nexus-snapshots-noplatform"/>
>         </chain>
>     </resolvers>
> </ivysettings>
> L.K.
> -----Original Message-----
> From: Zac Jacobson []
> Sent: Sunday, March 22, 2015 6:07 PM
> To:
> Subject: Re: cache busting and integration question
> If you set your resolver’s checkmodified flag to true, it will verify the
> timestamp of the published artifact and pull down a more recent version if
> the repo is newer than the cache. I’ve found that if you have chained
> resolvers, you need to do this at the top level and it affects all repos,
> not just your snapshot repository.
> Also, if you set the force attribute for your local resolver, it will
> always take a version from there rather than going to other resolvers in
> the chain. Once you’ve got changes published to your snapshot or release
> repo, you’ll want to clean that artifact out of your local repo.
> <
> >
> > On Mar 20, 2015, at 15:12, Loren Kratzke <>
> wrote:
> >
> > I am encountering challenges with the Ivy cache and also with setting up
> what I would consider to be a typical (if not classic) developer workflow.
> Here is the desired workflow:
> >
> > I have a Nexus repository (releases and snapshots) plus a local file
> system repository used for local development. I have Project which depends
> upon Library. I wish to modify Library, publish locally, and then pull it
> into the local build of Project.
> >
> > And for my Jenkins "continuous" integration server, if the copy of
> Library-1.0.0-SNAPSHOT in Nexus is newer than what is in the cache on the
> build machine, then I would like Ivy to grab the newer version (just like
> Maven does).
> >
> > ISSUE #1
> >
> > Assuming I start with an empty cache and empty internal repo on my local
> dev machine, I build Project and it pulls Library from Nexus. Perfect.
> Works great. But then if I publish Library locally and build Project again,
> I will get the cached version of Library that came from Nexus. No matter
> what I do, I will always get the cached copy. This happens even though my
> chain resolver specifies my internal repo before the Nexus repo.
> >
> > If I blow away the cache, then I get my locally published build, but
> only if I blow away the cache. Otherwise I get stale stuff. Issue #2 below
> is all about stale stuff.
> >
> > I debugged Ivy (for hours upon hours) and during the resolve, it checks
> my internal repo, finds the fresh artifact (yay!) and the next thing it
> does is checks its cache, finds a module descriptor that points at Nexus,
> and proceeds to pull in the wrong artifact into Project.
> >
> > The behavior is either a bug, or implies that Ivy assumes that an
> artifact can only come from one place, and if it came from there once, it
> will come from there forever, and will never change. Again, Ivy finds the
> artifact in local repo, stops searching, then ends up delivering the cached
> artifact that came from Nexus sometime earlier.
> >
> > ISSUE #2
> >
> > I have been working on this problem for quite some time and I thought I
> had it fixed, but I don't. That is, my integration server needs the latest
> snapshot build of Library-1.0.0-SNAPSHOT. But what I get is the cached
> version from the Ivy cache. An SVN commit happens, triggers a build of
> Library, Library get published, triggers a build of Project, and a stale
> build of Library is delivered to Project from the cache (which breaks
> Project build).
> >
> >
> > Ivy will resolve and cache deps, but once a dep is in the cache, that's
> what I get forever until I blow away the cache. I see this question asked
> quite often regarding obtaining latest integration versions, and I see that
> Ivy claims to support the local development workflow, but I have yet to see
> a working example of either of these. Surely some of you have solved these
> issues.
> >
> > Is there any place anywhere in the world that has an example of
> development workflow configuration and/or integration configuration? Can
> anybody provide an example? So far I have not found anything, and advice on
> the net seems to be general suggestions like "try this, try that" and
> nobody really has the configuration solution. Is Ivy capable of these
> behaviors or do I need to hand roll a dependency management solution. Maven
> is too strict for these projects (C++ code). Ivy is good if I can just
> obtain these fundamental behaviors.
> >
> > Note that I have not provided any configs in this message that
> demonstrate the issues I am having but definitely can upon request.
> >
> > Thanks in advance,
> >
> > L.K.
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message