ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Crahen" <eric.crahen.li...@gmail.com>
Subject Re: Opt out of the cache for a resolver
Date Sun, 24 Dec 2006 14:04:13 GMT
Caching is certainly useful for avoiding network I/O when possible, but
there is
more to a cache than copying a file somewhere on the local machine. Cache
consistency is equally important. A cache is not useful it contains the
wrong data.
This example is one use case where simply adding to a cache and never really
looking to see if you are caching the right things isn't enough for the
correct behavior.

Its interesting that you mention you'd like your cached copy of some
artifact to
still be available after its deleted. If I want to delete an artifact from
my repository
I wouldn't want people to be consuming it any longer. This could be part of
a path
to deprecation. I may notify my users that a package will soon be deprecated
and
simply remove it after fair warning. I may even want to add @Deprecated
annotations
to classes in a version of a package I plan to deprecate and do a
publication of the
same version of that package using overwrite="true". The path to deprecation
being
warning, package version that was warned about actually will begin to
generate
compiler warnings because I've republished with the annotations, and finally
removal.

I think for any local resolver checking the source and correctly
invalidating
the cache when you select useOrigin=true may be a reasonable behavior. Or
providing some other way for people who need this to be able to opt to have
a consistent cache when using remote resolvers.

Having a corrupt copy of an artifact in a cache is another use case where
the cache must be invalidated. This can happen in a number of ways.

---

That said, another way to look at this could be to ignore the caching aspect
for a moment. Step back and look at what I really want to do here.
Ultimately,
I'd like to publish a file to a local filesystem repo for overriding, and
then
be able to remove/unpublish that override w/o having to type a bunch of
commands or go looking for files in ivy's cache and the override - I really
don't
want people to go mucking with that.

One possible solution would be to make this erasing resolver that removes
files
for a package also in validate the cache. So to publish to this erasing
resolver
would mean please remove all artifacts and ivy files from the repository for

package "X" and also invalidate the cache for those versions.

The benefit here is that I can more effectively use this override
repository, and
I won't need to go tracking down files and deleting them manually (maybe I
get
ivyconf.xml from some else via a url and I don't know and shouldn't have to
know
where they configured the cache to be, maybe I also don't even know or care
the location of the local override. I could find out, but its not my job to
maintain
the ivyconf, thats someone elses role)

This would solve my particular use case for this local override repository.

However, you still have at least two other real use cases of cache
corruption
and deprecation that would be unsolved.


On 12/24/06, John Gill <llignhoj@gmail.com> wrote:
>
> I think the point is that the cache is there so that IVY doesn't have to
> check original repository were it came from (which could be remote, slow,
> offline, etc). In addition, how annoying would it be for you to have a
> copy
> of something in your cache, only to find it removed because it got deleted
> from some remote repository you were using.
>
> Personally, I think IVY is working they way it should, and wouldn't want
> it
> to check the remote repository to see if it needed to clean up the cache
> as
> that would make it slower.
>
> On 12/24/06, Eric Crahen <eric.crahen.lists@gmail.com> wrote:
> >
> > Here is a simple reproduction:
> >
> > I have two packages, Example1 and Example2, both just create and publish
> a
> > dummy jar
> > to a local filesystem resolver in ~/.ivy/repository
> >
> > Both use a copy of this build.xml snippet to interact with ivy.
> >
> > build.xml:
> >
> >     <ivy:resolve   useOrigin="true"/>
> >     <ivy:cachepath pathid="compile.classpath"
> >       useOrigin="true" conf="java.compile"/>
> >     <ivy:cachepath pathid="test.classpath"
> >       useOrigin="true" conf="java.test"/>
> >     <ivy:cachepath pathid="runtime.classpath"
> >       useOrigin="true" conf="java.runtime"/>
> >
> >     <ivy:deliver status="release"/>
> >
> >     <mkdir dir="${build.dir}/lib"/>
> >     <jar destfile="${build.dir}/lib/${ant.project.name}.jar"/>
> >
> >     <ivy:publish resolver="override"/>
> >
> > This is the relevant ivyconfig.xml shared by both:
> >
> > ivyconfig.xml:
> >
> >     <filesystem name="override" local="true"
> > alwaysCheckExactRevision="false">
> >       <ivy
> >         pattern="${local.repository.dir}/${ivy.pattern}" />
> >       <artifact
> >         pattern="${local.repository.dir}/${ivy.pattern}" />
> >     </filesystem>
> >
> > Example ivy.xml:
> >
> > <ivy-module version="1.0">
> >
> >   <info organisation="example" module="${ant.project.name}" revision="
> > 0.0.1"
> > status="integration"/>
> >
> >   <configurations>
> >     <conf name="java.compile" visibility="public"/>
> >     <conf name="java.test"    extends="java.compile"
> > visibility="private"/>
> >     <conf name="java.runtime" extends="java.compile"
> > visibility="private"/>
> >   </configurations>
> >
> >   <publications>
> >     <artifact name="${ant.project.name}" type="lib" ext="jar" conf="
> > java.runtime"/>
> >   </publications>
> >
> >   <dependencies>
> >   </dependencies>
> >
> > </ivy-module>
> >
> > Example2 ivy.xml:
> >
> > <ivy-module version="1.0">
> >
> >   <info organisation="example" module="${ant.project.name}" revision="
> > 0.0.1"
> > status="integration"/>
> >
> >   <configurations>
> >     <conf name="java.compile" visibility="public"/>
> >     <conf name="java.test"    extends="java.compile"
> > visibility="private"/>
> >     <conf name="java.runtime" extends="java.compile"
> > visibility="private"/>
> >   </configurations>
> >
> >   <publications>
> >     <artifact name="${ant.project.name}" type="lib" ext="jar" conf="
> > java.runtime"/>
> >   </publications>
> >
> >   <dependencies>
> >     <dependency org="example" name="Example" rev="0.0.1" conf="*->*"/>
> >   </dependencies>
> >
> > </ivy-module>
> >
> > ---
> >
> > Building Example1 results in this:
> >
> > cd ~/.ivy && find
> > ./repository
> > ./repository/example
> > ./repository/example/Example
> > ./repository/example/Example/0.0.1
> > ./repository/example/Example/0.0.1/ivy
> > ./repository/example/Example/0.0.1/ivy/ivy.xml.sha1
> > ./repository/example/Example/0.0.1/ivy/ivy.xml.md5
> > ./repository/example/Example/0.0.1/ivy/ivy.xml
> > ./repository/example/Example/0.0.1/lib
> > ./repository/example/Example/0.0.1/lib/Example.jar
> > ./repository/example/Example/0.0.1/lib/Example.jar.sha1
> > ./repository/example/Example/0.0.1/lib/Example.jar.md5
> > ./cache
> > ./cache/ivy-report.css
> > ./cache/example-Example-java.compile.xml
> > ./cache/example-Example-java.runtime.xml
> > ./cache/resolved-example-Example-0.0.1.properties
> > ./cache/example-Example-java.test.xml
> > ./cache/resolved-example-Example-0.0.1.xml
> > ./cache/ivy-report.xsl
> >
> > I see the artifacts published to my filesystem resolver because they
> > appear
> > under repository.
> > Now I build Example2
> >
> > cd ~/.ivy && find
> > ./repository
> > ./repository/example
> > ./repository/example/Example
> > ./repository/example/Example/0.0.1
> > ./repository/example/Example/0.0.1/ivy
> > ./repository/example/Example/0.0.1/ivy/ivy.xml.sha1
> > ./repository/example/Example/0.0.1/ivy/ivy.xml.md5
> > ./repository/example/Example/0.0.1/ivy/ivy.xml
> > ./repository/example/Example/0.0.1/lib
> > ./repository/example/Example/0.0.1/lib/Example.jar
> > ./repository/example/Example/0.0.1/lib/Example.jar.sha1
> > ./repository/example/Example/0.0.1/lib/Example.jar.md5
> > ./repository/example/Example2
> > ./repository/example/Example2/0.0.1
> > ./repository/example/Example2/0.0.1/ivy
> > ./repository/example/Example2/0.0.1/ivy/ivy.xml.sha1
> > ./repository/example/Example2/0.0.1/ivy/ivy.xml.md5
> > ./repository/example/Example2/0.0.1/ivy/ivy.xml
> > ./repository/example/Example2/0.0.1/lib
> > ./repository/example/Example2/0.0.1/lib/Example2.jar.sha1
> > ./repository/example/Example2/0.0.1/lib/Example2.jar.md5
> > ./repository/example/Example2/0.0.1/lib/Example2.jar
> > ./cache
> > ./cache/ivy-report.css
> > ./cache/example-Example-java.compile.xml
> > ./cache/example-Example-java.runtime.xml
> > ./cache/resolved-example-Example-0.0.1.properties
> > ./cache/example-Example-java.test.xml
> > ./cache/resolved-example-Example2-0.0.1.properties
> > ./cache/example-Example2-java.compile.xml
> > ./cache/resolved-example-Example-0.0.1.xml
> > ./cache/example-Example2-java.runtime.xml
> > ./cache/example
> > ./cache/example/Example
> > ./cache/example/Example/ivydata-0.0.1.properties
> > ./cache/example/Example/ivy-0.0.1.xml
> > ./cache/example-Example2-java.test.xml
> > ./cache/ivy-report.xsl
> > ./cache/resolved-example-Example2-0.0.1.xml
> >
> >
> > I see the artifacts published to my filesystem resolver because they
> > appear
> > under repository.
> > HOWEVER, I also see that my cache was populated with the descriptor from
> > the
> > first project.
> > -> ./cache/example/Example
> > -> ./cache/example/Example/ivydata-0.0.1.properties
> > -> ./cache/example/Example/ivy-0.0.1.xml
> >
> > Now when I remove Example from my repository and build Example2 again,
> Ivy
> > resolves
> > Example correctly, even though it has actually been removed.
> >
> > rm -r ~/.ivy/repository/example/Example
> >
> > It does this because the cache doesn't get invalidated. When the resolve
> > task runs during the
> > second build of Example2 in which I've removed the artifacts and ivy
> files
> > from the filesystem
> > repository by force, it seems to only consult the cache and actually
> says
> > "found [ example | Example
> > | 0.0.1 ] in cache" which doesn't make sense since I've specified
> > useOrigin=true.
> >
> > ----
> >
> > The use case for this is that I want to do something, like override
> > commons-http client locally to work
> > with a bugfix that hasn't been released yet. I use a chain with a local
> > filesystem resolver in the front
> > so that returnFirst="true". This way whatever I put in my local resolve
> > will
> > effectively override the
> > offical release. Now at some point I don't need the override - maybe the
> > bugfix is integrated into the
> > offical release - maybe I just decide that bug is not relevant to me
> > anymore
> > and I want to go back to
> > building against the offical copy. I want to just remove the local
> > override
> > by removing that artifact and
> > ivy file from my local filesystem. This fails because the cache is never
> > invalidated correctly.
> >
> > The work around is nuke the cache, which isn't ideal. Ultimately I'd
> like
> > to
> > never be typing rm to use
> > ivy effectively which is why this isn't ideal. I don't want to manually
> be
> > blasting things in ivy's cache.
> > Not having to manually blast things in my local repository is what the
> > https://issues.apache.org/jira/browse/IVY-358 request is for.
> >
> >
> >
> >
> > On 12/23/06, Eric Crahen <eric.crahen.lists@gmail.com> wrote:
> > >
> > > Is there a way I can fail to resolve a dependency if it does not
> appear
> > in
> > > the remote repository?
> > > For instance,
> > >
> > > If I resolve against a <url/> resolver, and I find the dependency
> there,
> > I
> > > download it, my cache is
> > > updated and I'll use the copy from my cache.
> > >
> > > Now if that dependency is removed from the <url/> resolver and I
> resolve
> > a
> > > second time, I fail to
> > > resolve the dependency, but I don't seem to invalidate the cache. So
> > even
> > > though the artifact no
> > > longer exists in the repository I will resolve it out of my cache.
> I'll
> > > even print a message like
> > > "found [ whatever ] in
> > > repository-that-acutally-no-longer-has-this-dependency"
> > >
> > >
> > > --
> > >
> > > - Eric
> >
> >
> >
> >
> > --
> >
> > - Eric
> >
> >
>
>
> --
> Regards,
> John Gill
>
>


-- 

- Eric

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message