ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Bodewig <bode...@apache.org>
Subject DirectoryScanner Memory Consumption
Date Fri, 19 Sep 2008 14:42:04 GMT
Hi all,

I've added a bunch of tests and I worked my way from Ant 1.7.1 to
trunk's HEAD by selective merging of the different changes.

Currently I don't think we'll get any closer to 1.7.1's memory usage
than we currently are (see below, the tests use the "big" setup of
dirscanner.xml).

I could pinpoint the increased memory usage to
FileUtils.isSymbolicLink.  trunk's HEAD contains this here

    private boolean causesIllegalSymlinkLoop(String dirName, File parent,
                                             LinkedList directoryNamesFollowed) {
        try {
            if (directoryNamesFollowed.size() >= maxLevelsOfSymlinks
                && CollectionUtils.frequency(directoryNamesFollowed, dirName)
                   >= maxLevelsOfSymlinks
                && FILE_UTILS.isSymbolicLink(parent, dirName)) {

and this is invoked for each directory that may be traversed if
followSymlinks==true.

If I change the code to read

                && CollectionUtils.frequency(directoryNamesFollowed, dirName)
                   >= maxLevelsOfSymlinks
                && false
                && FILE_UTILS.isSymbolicLink(parent, dirName)) {

I get the same memory usage as Ant 1.7.1 while changing it to

                && CollectionUtils.frequency(directoryNamesFollowed, dirName)
                   >= maxLevelsOfSymlinks
                && FILE_UTILS.isSymbolicLink(parent, dirName)
                && false) {

shows the same memory usage as trunk's HEAD (the body of the statement
is completely irrelevant).

Something in File.getCanonicalPath seems to consume memory that isn't
easily reclaimed by the GC, and I have no idea what we could do about
it.

I've tried JDK 1.6 instead of 1.4 and it helps a bit, memory usage is
closer to what I see in 1.7.1 (but not completely there).

Finally, the test case shows this effect stronger than most people
will see since the tree holds a lot of directories that have the same
name as at least one of their ancestors.  Real life trees probably
won't notice the change of memory consumption as much.

Stefan

Raw data:

svn rev 696674          introduction of VectorSet
svn rev 697032          removed memoization
svn rev 697073          removed LinkedList.clone()

Tests with [*] used <pathconvert>, other tests used <resourcecount>.
Tests with [+] use Java 1.6, other tests used Java 1.4.2

many-patterns
-------------

Ant 1.7.1        4 min 26 s  ~  39 MB [*]
svn rev 696674         29 s  ~ 179 MB [*]
svn rev 697032         27 s  ~  89 MB [*]
Ant 1.7.1        4 min 32 s  ~  38 MB
svn rev 696674         27 s  ~ 160 MB
svn rev 697032         25 s  ~  68 MB
svn rev 697073         25 s  ~  49 MB
svn rev 697073         25 s  ~  49 MB

Ant 1.7.1        3 min 22 s  ~  41 MB [+]
svn rev 697073         21 s  ~  55 MB [+]

many-roots
----------

Ant 1.7.1       15 min  6 s  ~ 204 MB [*]
svn rev 696674   1 min  1 s  ~ 350 MB [*]
svn rev 697032         49 s  ~ 310 MB [*]
Ant 1.7.1       15 min 25 s  ~  61 MB
svn rev 696674         43 s  ~ 236 MB
svn rev 697032         40 s  ~  99 MB
svn rev 697073         40 s  ~  98 MB

svn rev 697073         31 s  ~  73 MB [+]

matchall
--------

Ant 1.7.1       13 min 42 s  ~  67 MB [*]
svn rev 696674         49 s  ~ 249 MB [*]
svn rev 697032         47 s  ~ 275 MB [*]
Ant 1.7.1       13 min 34 s  ~  55 MB
svn rev 696674         39 s  ~ 235 MB
svn rev 697032         38 s  ~  97 MB
svn rev 697073         39 s  ~  97 MB

svn rev 697073         30 s  ~  73 MB [+]

name-matches
------------

Ant 1.7.1        3 min  1 s  ~  95 MB [*]
svn rev 696674         41 s  ~ 246 MB [*]
svn rev 697032         38 s  ~ 152 MB [*]
Ant 1.7.1        3 min 19 s  ~  52 MB
svn rev 696674         39 s  ~ 229 MB
svn rev 697032         36 s  ~  97 MB
svn rev 697073         35 s  ~  97 MB

Ant 1.7.1        2 min 22 s  ~  58 MB [+]
svn rev 697073         29 s  ~  72 MB [+]

recursive-excludes
------------------

Ant 1.7.1        1 min 55 s  ~  69 MB [*]
svn rev 696674         40 s  ~ 236 MB [*]
svn rev 697032         36 s  ~ 117 MB [*]
Ant 1.7.1        1 min 53 s  ~  49 MB
svn rev 696674         36 s  ~ 217 MB
svn rev 697032         34 s  ~  92 MB
svn rev 697073         34 s  ~  91 MB

Ant 1.7.1        1 min 27 s  ~  54 MB [+]
svn rev 697073         27 s  ~  63 MB [+]

roots
-----

Ant 1.7.1              11 s  ~  20 MB [*]
svn rev 696674          5 s  ~  35 MB [*]
svn rev 697032          4 s  ~  26 MB [*]
Ant 1.7.1              10 s  ~  14 MB
svn rev 696674          4 s  ~  29 MB
svn rev 697032          3 s  ~  20 MB
svn rev 697073          4 s  ~  20 MB

Ant 1.7.1               7 s  ~  20 MB [+]
svn rev 697073          3 s  ~  26 MB [+]

all
---

Ant 1.7.1                 not run
svn rev 696674   3 min 27 s  ~ 369 MB [*]
svn rev 697032   3 min  8 s  ~ 350 MB [*]
svn rev 696674   3 min 10 s  ~ 320 MB
svn rev 697032   2 min 53 s  ~ 138 MB
svn rev 697073   2 min 46 s  ~ 129 MB

svn rev 697073   2 min 18 s  ~  95 MB [+]


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org


Mime
View raw message