Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Message-ID: <3F70EE03.4080502@earthlink.net>
Date: Tue, 23 Sep 2003 19:06:11 -0600
From: Dmitry Serebrennikov <dmitrys@earthlink.net>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US;
 rv:1.3) Gecko/20030312
MIME-Version: 1.0
To: Lucene Developers List <lucene-dev@jakarta.apache.org>
Subject: Re: PATCH: IndexReaderDelete (Bugzilla Bug 12588)
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hello Otis,

I've been looking at the junit failure in the TestIndexReader and I 
think I know what's going on. Christoph's patches are, the right 
solution, but the intermittent failures that you've observed (and which 
I see as well) are due, I think, to the fact that timestamps from 
System.currentTimeMillis are only so accurate. I think that there is 
little enough code in the tests that the timestamp stored when an the 
index reader is opened is still the same as the one that the "segmnets" 
file is touched with!

This would probably also depend on the OS and even the clock of the 
hardware on which we are running! I'm running these tests in my JBuilder 
IDE on a Windows 2000 Pentium III. To confirm this, I added a 
Thread.sleep(x) statement into RAMDirectory's touchFile method. The 
following table shows the number of test failures in 20 consective loops 
of that test, while varying the x:
x      error count
--     -----------
0      18
1       11
2       8
5       5
10     0

How about we make the following change to the RAMDirectory's touchFile 
method. Of course the MONITOR stuff can be removed - it's there for 
further testing on other OSes. With this change, I believe Christoph's 
patches are ok to go in and they will fix the TestIndexReader failures.

  /** Set the modified time of an existing file to now. */
  public void touchFile(String name) throws IOException {
    final boolean MONITOR = false;
    int count = 0;
   
    RAMFile file = (RAMFile)files.get(name);
    long ts2, ts1 = System.currentTimeMillis();
    do {
        try {
            Thread.sleep(0, 1);
        } catch (InterruptedException e) {}
        ts2 = System.currentTimeMillis();
        if (MONITOR) count ++;
    } while(ts1 == ts2);
   
    file.lastModified = ts2;

    if (MONITOR)
        System.out.println("SLEEP COUNT: " + count);       
  }

When I run with this, I get 0 errors in 20 runs and I see that each run 
takes from 1 to 10 loops to get a unique time stamp. On average it 
appears to take about 6.

As far as the timestamps on the FSDirectory, it's probably expensive 
enough to change file's time as to not require any more delays. There's 
however another story to tell about the FSDirectory. When I switched to 
it, I started seeing other types of test failures. Some had to do with 
accessing a reader that has been closed - this does not fail with 
RAMDirectory. Other errors have to do with running the test repeatedly - 
some files remain open and can't be deleted. I'm working through this 
right now, but the point I wanted to make was that a large number of 
failures are not shown in test cases when running with RAMDirectory. I 
think we should try to run all test cases with both directories to 
really know that they pass.

More on this later.
Dmitry.