lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Rowe <>
Subject Re: Testcase failure on OSX
Date Tue, 30 Sep 2003 23:09:25 GMT
Why not just add 1000ms to the current time when touching the files 
(instead of waiting for a second to pass).  I tried in on my Linux box 
(e2fs) via the java File.setLastModified() method, and it allowed the 
file to have a timestamp in the future.

Wouldn't this be a much smaller effort?


Hani Suleiman wrote:

> Aha, yeah, System.currentTimeMillis would definitely be of much higher 
> resolution.
> I know the problem also intermittently affects linux, as I've seen it 
> happen on there too (but a lot more rarely).
> I think it's a feature of the antiquated HFS mac filesystem that OSX 
> yses by default, rather than the OS itself. Adding a long timestamp to 
> the segments file would fix this everywhere I think, since the worst 
> resolution timers out there are in the region of 10ms, which isn't so bad.
> I can work on that (pretty big, I suspect) patch if everyone else is too 
> busy, since I'm quite looking forward to all tests passing on all 
> platforms. I can verify on windows, linux, solaris, and OSX. Of course 
> it'd be better if someone a bit more experienced with the codebase were 
> to handle it ;)
> On Tuesday, September 30, 2003, at 06:37 PM, Dmitry Serebrennikov wrote:
>> Hani Suleiman wrote:
>>> The count for FSDirectory on OSX is around 700-800. The problem is 
>>> that  the filesystem timestamps on OSX has abysmal resolution 
>>> (nothing finer  than 1 whole second!)
>> Wow! Write once, debug everywhere...
>> Is this an OSX "feature" or is this a feature of a particular disk / 
>> filesystem / driver / whatnot that is in use in the current run?
>> If this is unavoidable on OSX, perhaps other platforms are also 
>> affected? Does anyone know the timestamp resolution on Wintel, Lintel, 
>> Solaris, etc?
>> Finally, what if we allocate a "long" in the beginning of the files 
>> that need to be touched (like the segments and other files) and use it 
>> to store the full System.currentTimeMillis timestamp? Does this method 
>> return a more granular number on OSX? This would be a big change, but 
>> it would work the same on all platforms and will not have a slowdown.
>>> So the touch method will spin its wheels for anything up to a second  
>>> before we've registered an actual 'touch'.
>>> This isn't so bad as files aren't touch that often, but it's a lot 
>>> more  problematic in the case of SegmentInfos.write(), where a new 
>>> file is  created, then copied over the old one. The  
>>> testDeleteReaderWriteConflict testcase fails due to this. The old 
>>> file  and the new have the same timestamp, and so the staleness check 
>>> fails.  How should this be fixed?
>>> Also, won't the fix for FSDirectory mean there's a window where other 
>>>  threads could end up with bad info, if we're taking up to a second 
>>> to  touch a file?
>> I don't know off hand, but I believe there is file locking in place to 
>> prevent other threads from getting into this code until the update 
>> operation is completed, and it will not be completed until the touch 
>> finishes. This still sucks with a 1-second blocking...
>> Dmitry.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Steve Rowe
Software Engineer
Center for Natural Language Processing
School of Information Studies
Syracuse University

View raw message