commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bojan Vukojevic (JIRA)" <j...@apache.org>
Subject [jira] Created: (TRANSACTION-26) applyDeletes in commons-transaction-1.2 is very slow in case when large number of files needs to be deleted from the directory
Date Wed, 05 Sep 2007 22:05:47 GMT
applyDeletes in commons-transaction-1.2 is very slow in case when large number of files needs
to be deleted from the directory
------------------------------------------------------------------------------------------------------------------------------

                 Key: TRANSACTION-26
                 URL: https://issues.apache.org/jira/browse/TRANSACTION-26
             Project: Commons Transaction
          Issue Type: Improvement
    Affects Versions: 1.2
         Environment: Linux/JDK1.5
            Reporter: Bojan Vukojevic
             Fix For: 1.2


method applyDeletes in FileResourceManager lists the directory after every file delete. In
case when there is a large number of files in the directory that needs to be deleted, this
can be very slow. I had a case where directory had ~100,000.00 files. It took 5 hours for
this method to complete. Simple improvement can be done where the check for the empty directory
is moved outside of the loop that is deleting the files, i.e.:

Original:

protected static void applyDeletes(File removeDir, File targetDir, File rootDir) throws IOException
{
        if (removeDir.isDirectory() && targetDir.isDirectory()) {
            File[] files = removeDir.listFiles();
            for (int i = 0; i < files.length; i++) {
                File removeFile = files[i];
                File targetFile = new File(targetDir, removeFile.getName());
                if (removeFile.isFile()) {
                    if (targetFile.exists()) {
                        if (!targetFile.delete()) {
                            throw new IOException("Could not delete file " + removeFile.getName()
                                    + " in directory targetDir");
                        }
                    }
                    // indicate, this has been done
                    removeFile.delete();
                } else {
                    applyDeletes(removeFile, targetFile, rootDir);
                }
                // delete empty target directories, except root dir
                if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
                    targetDir.delete();
                }
            }
        }
    }

=======================
Much Faster:
protected static void applyDeletes(File removeDir, File targetDir, File rootDir) throws IOException
{
        if (removeDir.isDirectory() && targetDir.isDirectory()) {
            File[] files = removeDir.listFiles();
            for (int i = 0; i < files.length; i++) {
                File removeFile = files[i];
                File targetFile = new File(targetDir, removeFile.getName());
                if (removeFile.isFile()) {
                    if (targetFile.exists()) {
                        if (!targetFile.delete()) {
                            throw new IOException("Could not delete file " + removeFile.getName()
                                    + " in directory targetDir");
                        }
                    }
                    // indicate, this has been done
                    removeFile.delete();
                } else {
                    applyDeletes(removeFile, targetFile, rootDir);
                }                
            }
         // delete empty target directories, except root dir
            if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
                targetDir.delete();
            }
        }
    }


Below is the patch generated based on the released 1.2 version:

Index: //10.1.31.20/bvukojev/DS7.1_EclipseWorkspace/asf-commons-transaction-1.2/src/java/org/apache/commons/transaction/file/FileResourceManager.java
===================================================================
--- //10.1.31.20/bvukojev/DS7.1_EclipseWorkspace/asf-commons-transaction-1.2/src/java/org/apache/commons/transaction/file/FileResourceManager.java
(revision 573053)
+++ //10.1.31.20/bvukojev/DS7.1_EclipseWorkspace/asf-commons-transaction-1.2/src/java/org/apache/commons/transaction/file/FileResourceManager.java
(working copy)
@@ -162,12 +162,12 @@
                     removeFile.delete();
                 } else {
                     applyDeletes(removeFile, targetFile, rootDir);
-                }
-                // delete empty target directories, except root dir
-                if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
-                    targetDir.delete();
-                }
+                }                
             }
+         // delete empty target directories, except root dir
+            if (!targetDir.equals(rootDir) && targetDir.list().length == 0) {
+                targetDir.delete();
+            }
         }
     }
 










-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message