ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Lalevée <nicolas.lale...@hibnet.org>
Subject A performance issue in 'restrict'
Date Mon, 14 Jun 2010 17:28:17 GMT
When working on IvyDE's build to get dependencies from Eclipse mirrors, I tried to use resourcelist,
as documented in the examples of the doc [1].

But it was very slow because Ant is checking every urls and some mirrors are not very responsive.
So I had to workaround and filter the urls to select only the first one and hoping it is available:

<copy todir="${basedir}">
    <first>
        <restrict>
            <resourcelist>
                <!-- get the xml mirror list -->
                <url url="${hudson.download.baseurl}?file=@{dropdir}/@{file}&amp;protocol=http&amp;format=xml"
/>
                <filterchain> <!-- change the xml into flat url list -->
                    <linecontainsregexp>
                        <regexp pattern="^.*mirror url=&quot;([^&quot;]*)&quot;.*$"
/>
                    </linecontainsregexp>
                    <tokenfilter>
                        <replaceregex pattern="^.*mirror url=&quot;([^&quot;]*)&quot;.*$"
replace="\1" flags="gi" />
                    </tokenfilter>
                    <headfilter lines="1" /> <!-- HACK to force Ant to not check
every urls -->
                </filterchain>
            </resourcelist>
            <exists/> <!-- restrict to only responsive mirror--> 
        </restrict>
    </first> <!-- copy the first responsive one -->
    <flattenmapper/> <!-- avoid the useless creation of folders-->
</copy>

I looked into the code, it seems org.apache.tools.ant.types.resources.Restrict is the culprit.
It uses BaseResourceCollectionWrapper which loads the entire underlying resource collection.
I searched for the use of BaseResourceCollectionWrapper in Ant and I think that this performance
issue may also affect 
org.apache.tools.ant.types.resources.SizeLimitCollection and org.apache.tools.ant.types.resources.Tokens.

I started to think about a fix but it seems non trivial. If I'm not mistaken we would have
to implement an iterator which should deal with an optionnal cache and concurrency.
I am also worried about the isFilesystemOnly implementation in Restrict. Should it return
true if actually selected resources are files, or return true if the entire set of candidates
are files ? In the first case it would make the iterator useless.

Side note: I didn't checked that piece of script in as our Hudson instance doesn't have Ant
1.8 installed. Not yet. I'll ask for it.

Nicolas

[1] http://ant.apache.org/manual/Types/resources.html#resourcelist

PS: I started recently to intensively use loadresource and all the resource related stuff
in my build scripts, it is amazingly porwerful ! It reminds me when I was 'pipelining' in
cocoon ;)


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org


Mime
View raw message