lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hani Suleiman <h...@formicary.net>
Subject Re: Testcase failure on OSX
Date Tue, 30 Sep 2003 21:27:56 GMT
The count for FSDirectory on OSX is around 700-800. The problem is that  
the filesystem timestamps on OSX has abysmal resolution (nothing finer  
than 1 whole second!)

So the touch method will spin its wheels for anything up to a second  
before we've registered an actual 'touch'.

This isn't so bad as files aren't touch that often, but it's a lot more  
problematic in the case of SegmentInfos.write(), where a new file is  
created, then copied over the old one. The  
testDeleteReaderWriteConflict testcase fails due to this. The old file  
and the new have the same timestamp, and so the staleness check fails.  
How should this be fixed?

Also, won't the fix for FSDirectory mean there's a window where other  
threads could end up with bad info, if we're taking up to a second to  
touch a file?

On Tuesday, September 30, 2003, at 01:12 PM, Dmitry Serebrennikov wrote:

> Sounds like the same error that I fixed in the RAMDirectory last week.  
> I didn't think that FSDirectory would have this problem because I  
> thought that the OS filesystem would take care of this. Apparently it  
> does not.
>
> Hani, please take a look at the RAMDirectory touchFile method (line  
> 148). It's basically the same fix you proposed except that it has some  
> commented out monitoring code and it uses Thread.sleep(0,1) to release  
> the CPU while waiting. In my measurements I was seeing 5-10 loops with  
> the Thread.sleep call (regardless of the actual sleep duration that I  
> specified), and around 10000 loops without a sleep call. But of course  
> I was on a different hardware/OS combo and probably a different VM.
>
> Hani, will you be able to try a variant of this fix for the  
> FSDirectory?
> I'm currious to see what kinds of results will observed on OS X.
>
> Thanks.
> Dmitry.
>
>
> Hani Suleiman wrote:
>
>> Ok, I figured out what's going on.
>>
>> I suspect this will only manifest itself on faster machines (I'm on a  
>>  dual 1.42ghz G4)
>>
>> What happens basically is that the the timer resolution on   
>> file.lastModified isn't accurate enough, so when a file is touched,  
>> it  returns the same value for lastModified. This in turn causes the   
>> segment staleness checks in IndexReader to fail (around line 276,   
>> lastModified(directory)==segmentInfosAge), hence the testcase fails.
>>
>> I believe this is a bug, but I can't come up with a satisfactory   
>> workaround. My solution is to modify FSDirectory.touchFile(String  
>> name)  to have the following block;
>>
>>     long oldModified = file.lastModified();
>>     do
>>     {
>>       file.setLastModified(System.currentTimeMillis());
>>     }
>>     while(oldModified==file.lastModified());
>>
>> This ensures that a touch ensures the file is considered 'fresher'  
>> that  it was before, which makes everything happy. It's a bit of a  
>> hack  though, so I'm not sure of how you guys would prefer handling  
>> this,  take the performance hit with the extra lastModified checks,  
>> if make it  an even bigger hack by checking for OSX, or something far  
>> more clever  and not detrimental to performance ;)
>>
>> Hani
>>
>> On Tuesday, September 30, 2003, at 12:42 AM, Hani Suleiman wrote:
>>
>>> here you go, happens to me on linux too (with JDK 1.4.0)
>>>
>>> OSX using latest 1.4.1
>>>
>>>    [junit] ------------- ---------------- ---------------
>>>     [junit] Testsuite: org.apache.lucene.index.TestIndexReader
>>>     [junit] Tests run: 5, Failures: 2, Errors: 0, Time elapsed:  
>>> 2.883  sec
>>>
>>>     [junit] Testcase:   
>>> testDeleteReaderWriterConflict(org.apache.lucene.index.TestIndexReade 
>>> r) :  FAILED
>>>     [junit] Delete allowed on an index reader with stale segment   
>>> information
>>>     [junit] junit.framework.AssertionFailedError: Delete allowed on  
>>> an  index reader with stale segment information
>>>     [junit]     at   
>>> org.apache.lucene.index.TestIndexReader.testDeleteReaderWriterConflic 
>>> t( TestIndexReader.java:271)
>>>     [junit]     at  
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native  Method)
>>>     [junit]     at   
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. 
>>> ja va:39)
>>>     [junit]     at   
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces 
>>> so rImpl.java:25)
>>>
>>>
>>>     [junit] Testcase:   
>>> testDeleteReaderReaderConflict(org.apache.lucene.index.TestIndexReade 
>>> r) :  FAILED
>>>     [junit] Delete allowed from a stale index reader
>>>     [junit] junit.framework.AssertionFailedError: Delete allowed  
>>> from  a stale index reader
>>>     [junit]     at   
>>> org.apache.lucene.index.TestIndexReader.testDeleteReaderReaderConflic 
>>> t( TestIndexReader.java:397)
>>>     [junit]     at  
>>> sun.reflect.NativeMethodAccessorImpl.invoke0(Native  Method)
>>>     [junit]     at   
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. 
>>> ja va:39)
>>>     [junit]     at   
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces 
>>> so rImpl.java:25)
>>>
>>>
>>>
>>> On Tuesday, September 30, 2003, at 12:37 AM, Erik Hatcher wrote:
>>>
>>>> Hani,
>>>>
>>>> I don't see this test failure on OS X.  Thanks to your patch on the  
>>>>  RussianAnalyzer, all my tests now successfully pass.  Could you  
>>>> give  us the stack trace for the problem you're seeing?
>>>>
>>>>     Erik
>>>>
>>>>
>>>> On Monday, September 29, 2003, at 11:28  PM, Hani Suleiman wrote:
>>>>
>>>>> I'm seeing a fairly disturbing testcase failure on OSX in   
>>>>> TestIndexReader.testDeleteReaderWriterConflict and   
>>>>> TestIndexReader.testDeleteReaderReaderConflict
>>>>>
>>>>> This only happens on OSX though, is this a known issue? Any have  
>>>>> an  ideas of what sort of thing is going wrong before I dig in  
>>>>> further?
>>>>>
>>>>> Hani
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------- 
>>>>> --
>>>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------- 
>>>> -
>>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>


Mime
View raw message