ant-ivy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Damm <bd...@silverspringnet.com>
Subject Re: Speeding up ivy:resolve
Date Thu, 20 Nov 2008 19:42:46 GMT
Hi Maarten,

  For improving resolve's performance I was thinking of caching artifact->location data.
 From what I can tell the 
slowness is in scanning the filesystem for matching packages.  Part of the responsability
for the slowness may be in 
having too many matching rules in the filesystem resolver.  Once the resolver has run and
located all of the 
dependencies, I was hoping to cache a simple map of the artifacts to their locations.  Then,
on another call to resolve, 
resolve can check if this particular dependency calculation has been resolved and cached and
if so just use those 
locations with no further computation. 

Another task would be needed to clear the cache (perhaps just delete the file in which the
location cache is stored).

Like I said, some of the responsability may be in having too many rules:

	  <!-- general resolver for pulling artifacts. -->
      <filesystem name="ivycontrib">
		 <!-- critical -->
         <artifact pattern="${ivycontrib.dir}/m2/[organisation]/[module]/[revision]/[type]s/[artifact]-[revision].
[ext]" />
         <artifact pattern="${ivycontrib.dir}/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"
/>
		 <!-- our custom packages -->
         <artifact pattern="${ivycontrib.dir}/[organisation]/[module]/[revision]/[artifact].[ext]"
/>

         <!-- we need to obsolete these next three -->
         <artifact pattern="${ivycontrib.dir}/[module]/[revision]/[artifact]-[revision].[ext]"
/>
         <artifact pattern="${ivycontrib.dir}/[module]/[revision]/[artifact]-${jdkXX}-[revision].[ext]"
/>
         <artifact pattern="${ivycontrib.dir}/[module]/[revision]-${os.name}-${os.arch}/[artifact].[ext]"
/>
         <artifact pattern="${ivycontrib.dir}/[module]/[revision]/[artifact].[ext]" />
         <!-- we need to obsolete these next two -->
         <artifact pattern="${ivycontrib.dir}/[organisation]-[module]/[revision]/[artifact].[ext]"
/>
         <artifact pattern="${ivycontrib.dir}/[organisation]-[module]/[revision]/[artifact]-[revision].[ext]"
/>

         <ivy pattern="${ivycontrib.dir}/m2/[organisation]/[module]/[revision]/ivy.xml"
/>
         <ivy pattern="${ivycontrib.dir}/[organisation]/[module]/[revision]/ivy.xml" />
         <ivy pattern="${ivycontrib.dir}/[module]/[revision]/ivy.xml" />
         <ivy pattern="${ivycontrib.dir}/[organisation]-[module]/[revision]/ivy.xml" />
         <ivy pattern="${ivycontrib.dir}/[module]/[revision]-${os.name}-${os.arch}/ivy.xml"
/>
      </filesystem>

-Ben

On Wednesday 19 November 2008 02:57:39 pm Maarten Coene wrote:
> Benjamin,
>
> could you give a bit more info about what your caching facade will do and
> how you think it will improve the resolve performance?
>
> Maarten
>
>
>
>
> ----- Original Message ----
> From: Benjamin Damm <bdamm@silverspringnet.com>
> To: ivy-user@ant.apache.org
> Cc: Colin Fleming <colin.mailinglist@gmail.com>
> Sent: Wednesday, November 19, 2008 9:21:57 PM
> Subject: Re: Speeding up ivy:resolve
>
> Good stuff and good tip on the getHostName() call, could be a really simple
> win there.  For the main work it'll be a few weeks before I can start on it
> but it's great to know there may be opportunities for shared effort
> especially in testing & bulletproofing any new code.
>
> Cheers!
>
> On Wednesday 19 November 2008 11:12:34 am Colin Fleming wrote:
> > Hi Ben,
> >
> > I'd be interested, for sure. We have about 190 modules in our project now
> > and resolve is a large part of our build time. I've been meaning to look
> > more closely at it for ages now, but I'm totally swamped.
> >
> > One thing I did see when I profiled our build a while back was that
> > somewhere in the depths there (I can't remember the exact spot, sorry)
> > Ivy calls getHostName() and doesn't cache the return value. This ends up
> > being called a lot of times (I don't remember the exact number but IIRC
> > in the thousands) due to recursive resolves, and would be a two-line fix.
> > I suspect this is the cause of a wide variation in our build times across
> > different machines that we're seeing (particularly that builds take much
> > longer in our integration server).
> >
> > If I get a moment I'll profile our build again (easy because it's written
> > in Java) and post any relevant info here.
> >
> > Cheers,
> > Colin
> >
> > 2008/11/19 Benjamin Damm <bdamm@silverspringnet.com>
> >
> > > Hi,
> > >
> > >  Does anyone have or would anyone be interested in a caching facade to
> > > speed
> > > up ivy:resolve?  I have a case where my project has about 100 modules
> > > in a few different configurations and it is taking a little longer to
> > > do the resolve than I can stand.  I'm concerned about what will happen
> > > when I introduce NFS into the mix.  For these reasons I'm considering
> > > what options I
> > > may have for caching the ivy:resolve (if this is possible) by perhaps
> > > storing
> > > structures to disk in a file that can be updated (perhaps by deleting
> > > it, I'm
> > > not sure yet).
> > >
> > >  The difference between this cache and the ivy cache is I'm not copying
> > > the files out of their locations; where they are presently is just fine
> > > because the entire repository is on local disk (and managed by p4).  So
> > > really this comes down to speeding up the filesystem resolver.
> > >
> > > -Ben
> > >
> > > --
> > > Benjamin Damm
> > > Silver Spring Networks
> > > 650-298-4200 x201



-- 
Benjamin Damm
Silver Spring Networks
650-298-4200 x201

Mime
View raw message