lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Isakson" <Eric.Isak...@sas.com>
Subject RE: Problem with long run IndexSearcher
Date Thu, 22 May 2003 13:56:40 GMT
I had to build it in the lucene package hierarchy because the methods on IndexReader that I'm
extending are not public or protected, they are package scoped.

Unless the lucene jar is sealed, you should be able to add this class to some other directory
or jar in your classpath that has the correct package structure without having to recompile
the whole of Lucene.

We saw a few of these go away in the move from Lucene 1.2 to 1.3 where there were some visibilty
changes to a few classes. IndexReader was not one of them.

I've been thinking a bit about my implementation, I don't think it would be safe to call delete
on this IndexReader since the document number used when doing a search could change if the
reader delegate is closed and reopened between the search that got the document number and
the time delete is called (this is also the reason it could give you inconsistent search results).
It should probably throw a NotSupportedException on calls to delete. It should probably also
be modified so it you can tell it to only refresh when it is idle (although on a busy system
that might make using it's refresh functionality somewhat pointless).

Eric

-----Original Message-----
From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com] 
Sent: Thursday, May 22, 2003 4:04 AM
To: Lucene Developers List
Subject: Re: Problem with long run IndexSearcher


Eric,

I have start trying to integrate your class in our application. I have tried to move the class
out of the lucene package to be able to  
use it without needing to recompile the whole lucene package.

I get this error:
src/java/com/extrapola/search/ 
IdleTimeoutRefreshingIndexReader.java:400: directory is not public in  
org.apache.lucene.index.IndexReader; cannot be accessed from outside  
package
                                         synchronized (directory) { //  
in- & inter-process sync

As far as I can understand it, this means that your code can not be  
inserted in an external package due to limited visibility granted to  
the directory attribute in the lucene code.

Is this correct? Is there any reason to avaid this kind of extensions  
in other packages?

Thanks for your attention,

Giulio Cesare Solaroli

On Monday, May 19, 2003, at 18:03 Europe/Rome, Eric Isakson wrote:

> Hope it works like I planned it to :-) If you have any performance
> metrics, I'd be interested to see how much it affects your search  
> performance.
>
> Eric
>
> -----Original Message-----
> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
> Sent: Monday, May 19, 2003 10:59 AM
> To: Lucene Developers List
> Subject: Re: Problem with long run IndexSearcher
>
>
> Eric,
>
> On Monday, May 19, 2003, at 16:52 Europe/Rome, Eric Isakson wrote:
>
>> I was just reading Karsten's note, I'm not sure what impact my 
>> strategy will have if your application pages through hit results over 
>> a period of time...might not work at all (or perhaps inconsistently). 
>> The application I use recalculates search  results for each page as I 
>> page through hits rather than trying to hold onto the results from 
>> page to page. If the reader changed to an index that resulted in 
>> different hits it would seem inconsistent. You would only notice the 
>> inconsistency if you happened to be paging around while my reader was 
>> being refreshed :(
>
> At the moment we re-run the query to page through a large result set. 
> This could cause some inconsistency in the results, but the problem is 
> really marginal in our environment.
>
>> Hmph, guess I didn't really solve the problem with this new class, 
>> but it is a start on a solution and I'd be happy to keep working.
>
> For us it could be a definite solution, even if I think that a more 
> elegant solution could give Lucene a final boost in becoming the 
> ultimate solution for searching!! ;-]
>
> Giulio Cesare
>
>
>> -----Original Message-----
>> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
>> Sent: Monday, May 19, 2003 10:44 AM
>> To: Lucene Developers List
>> Subject: Re: Problem with long run IndexSearcher
>>
>>
>> Hi Eric,
>>
>> On Monday, May 19, 2003, at 16:18 Europe/Rome, Eric Isakson wrote:
>>
>>> bah, accidently sent that before I was finished typing...
>>>
>>> see:
>>> http://nagoya.apache.org/eyebrowse/ReadMsg?listName=lucene-
>>> user@jakarta.apache.org&msgNo=1859
>>>
>>> which really should be added to the FAQ.
>>>
>>> It just so happens that I spent a good deal of my weekend working on 
>>> a class to solve this problem. It hasn't been tested much and I'm 
>>> new to writing apps that deal with threading issues (so I don't know 
>>> if I blundered anywhere). I'd love to see if it works and get 
>>> feedback from anyone that uses it.
>>
>> Great!!!
>>
>>> See my message at:
>>>
>>> http://nagoya.apache.org/eyebrowse/ReadMsg?listName=lucene-
>>> dev@jakarta.apache.org&msgNo=3393
>>
>> I did read it, but not carefully enught to realize it was what I was 
>> looking for. I will give it a try!!
>>
>>> If you have any feedback, pleass attach comments to the bug.
>>
>> Ok. I will let you know.
>>
>> Thanks again for your support.
>>
>> Giulio Cesare
>>
>>
>>
>>> Eric
>>>
>>> --
>>> Eric D. Isakson        SAS Institute Inc.
>>> Application Developer  SAS Campus Drive
>>> XML Technologies       Cary, NC 27513
>>> (919) 531-3639         http://www.sas.com
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Eric Isakson [mailto:Eric.Isakson@sas.com]
>>> Sent: Monday, May 19, 2003 10:05 AM
>>> To: Lucene Developers List
>>> Subject: RE: Problem with long run IndexSearcher
>>>
>>>
>>> Giulio,
>>>
>>> This problem stems from the fact that the IndexReader won't see your 
>>> changed index.
>>>
>>> See the faq entries:
>>>
>>>
>>> -----Original Message-----
>>> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
>>> Sent: Monday, May 19, 2003 9:38 AM
>>> To: lucene-dev@jakarta.apache.org
>>> Subject: Problem with long run IndexSearcher
>>>
>>>
>>> Hi all,
>>>
>>> first let me express my compliments for Lucene.
>>> I have been up for a full week-end to double check the results I was 
>>> having because I couldn't belive what I saw; with a stupid 
>>> application I could index DB data at a sustained rate of 50 
>>> documents per second.
>>>
>>> Now we have more that 2 millions documents indexed and the 
>>> performance are still excellent; our main bottle neck is still the 
>>> DB.
>>>
>>> Our situation:
>>> - we are indexing new documents at a sustained rate (an average of 
>>> 40.000 new documents a day);
>>> - we have written a small xmlRpc server in Java to search the index 
>>> from other applications.
>>>
>>> The xmlRpc server creates a single instance of IndexSearcher a reuse 
>>> it for each query issued. For each request, a new Query object is 
>>> created and the documents found are returned to the client.
>>>
>>> The problem we are seeing is that the documents indexed after the 
>>> xmlRpc server is started will not be found until the server is 
>>> restarted.
>>>
>>> Is this our foult, or the way IndexSearcher should work?
>>>
>>> What is the best way to keep the IndexSearcher up to date with the 
>>> updated index?
>>>
>>> Thanks for your attention,
>>>
>>> Giulio Cesare Solaroli
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message