Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Date: Wed, 22 Oct 2003 22:37:58 -0400
Mime-Version: 1.0 (Apple Message framework v552)
Content-Type: text/plain; charset=US-ASCII; format=flowed
Subject: timestamp timings
From: Hani Suleiman <hani@formicary.net>
To: lucene-dev@jakarta.apache.org
Content-Transfer-Encoding: 7bit
Message-Id: <E9EC314B-0501-11D8-895E-000A956D3476@formicary.net>

I've run this test and listing files is pretty consistently faster than 
opening and reading a file. The list approach becomes slower than the 
file when there are around 20+ files in the directory, so if a solution 
whereby timestamps are put in a dedicated dir is put in place, it's 
consistently faster than reading a file. I'm including my micro 
benchmark inline, so feel free to run it and improve either time ;)

For writing, the stamp-in-name solution is faster (obviously) since 
we're just creating an empty file, vs creating a file and writing into 
it.

Also I think that holding the file open is probably a bad idea, given 
that it'd rule out multiple VM's using the same index.

public class FileTest
{
   static File dir = new File("testdir");
   static class TimestampFileFilter implements FileFilter
   {
     private String name;

     TimestampFileFilter(String name)
     {
       this.name = name;
     }

     public boolean accept(File pathname)
     {
       return pathname.getName().startsWith(name + ".");
     }
   }

   public static long testFileList(String name)
   {
     TimestampFileFilter filter = new TimestampFileFilter(name);
     File[] files = dir.listFiles(filter);
     long latest = 0;
     for(int i = 0; i < files.length; i++)
     {
       File timestampFile = files[i];
       String fileName = timestampFile.getName();
       long timestamp = 
Long.parseLong(fileName.substring(fileName.lastIndexOf('.') + 1));
       if(timestamp > latest) latest = timestamp;
     }
     return latest;
   }

   public static long testReadFile(File file) throws IOException
   {
     FileInputStream fis = new FileInputStream(file);
     DataInputStream dis = new DataInputStream(fis);
     long timestamp = dis.readLong();
     fis.close();
     return timestamp;
   }

   public static void main(String[] args) throws IOException
   {
     dir.mkdir();
     File file = new File(dir, "blah.1234");
     FileOutputStream fos = new FileOutputStream(new File(dir, 
"timestamp"));
     DataOutputStream dos = new DataOutputStream(fos);
     dos.writeLong(System.currentTimeMillis());
     fos.close();
     file.createNewFile();
     long now = System.currentTimeMillis();
     int iterations = 100000;
     for(int i=0;i<iterations;i++)
     {
       testFileList("blah");
     }
     System.out.println("list time=" + 
((double)(System.currentTimeMillis()-now)/iterations) + "ms");
     for(int i = 0; i < iterations; i++)
     {
       testReadFile(new File(dir, "timestamp"));
     }
     System.out.println("read time=" + 
((double)(System.currentTimeMillis() - now) / iterations) + "ms");
   }
}


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org