lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Harwood <markharw...@yahoo.co.uk>
Subject Re: Improving TimeLimitedCollector
Date Sat, 27 Jun 2009 00:31:18 GMT
Going back to my post re TimeLimitedIndexReaders - here's an  
incomplete but functional prototype:

http://www.inperspective.com/lucene/TimeLimitedIndexReader.java
http://www.inperspective.com/lucene/TestTimeLimitedIndexReader.java


The principle is that all reader accesses check a volatile variable  
indicating something may have timed out (no need to check thread  
locals etc.) If and only if a time out has been noted threadlocals are  
checked to see which thread should throw a timeout exception.

All time-limited use of reader must be wrapped in try...finally calls  
to indicate the start and stop of a timed set of activities. A  
background thread maintains the next anticipated timeout deadline and  
simply waits until this is reached or the list of planned activities  
changes with new deadlines.


Performance seems reasonable on my Wikipedia index:

//some tests for heavy use of termenum/term docs
Read term docs for 200000 terms  in 4755 ms using no timeout limit  
(warm up)
Read term docs for 200000 terms  in 4320 ms using no timeout limit  
(warm up)
Read term docs for 200000 terms  in 4320 ms using no timeout limit
Read term docs for 200000 terms  in 4388 ms using  reader with time- 
limited access

//Example query with heavy use of termEnum/termDocs
+text:f* +text:a* +text:b* no time limit matched 1090041 docs in 2000 ms
+text:f* +text:a* +text:b* time limited collector matched 1090041 docs  
in 1963 ms
+text:f* +text:a* +text:b* time limited reader matched 1090041 docs in  
2121 ms

//Example fuzzy match burning CPU reading TermEnum
text:accomodation~0.5 no time limit matched 192084 docs in 	6428 ms
text:accomodation~0.5 time limited collector matched 192084 docs in 	 
5923 ms
text:accomodation~0.5 time limited reader matched 192084 docs in 	5945  
ms


The reader approach to limiting time is slower but has these  
advantages :

1) Multiple reader activities can be time-limited rather than just  
single searches
2) No code changes required to scorers/queries/filters etc
3) Tasks that spend plenty of  time burning CPU before collection  
happens can be killed earlier

I'm sure there's some thread safety issues to work through in my code  
and not all reader classes are wrapped (e.g. TermPositions) but the  
basics are there and seem to be functioning

Thoughts?
Mime
View raw message