lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giulio Cesare Solaroli <slrg...@ibn-italy.com>
Subject Re: Problem with long run IndexSearcher
Date Thu, 22 May 2003 14:14:43 GMT
Eric,

thanks for your suggestion; I am not a great Java programmer (as I 
think you can easily understand).
Thank for the suggestion. I will try to add a lucene like structure to 
my source tree to see if it can safely compile.

I will keep you updated on my progress. As soon as I can compile it, I 
will also look a little bit closer to the code itself and see if I can 
make any improvement/suggestions based on our (at the moment short and 
limited) experience with lucene.

Regards,

Giulio Cesare

On Thursday, May 22, 2003, at 15:56 Europe/Rome, Eric Isakson wrote:

> I had to build it in the lucene package hierarchy because the methods 
> on IndexReader that I'm extending are not public or protected, they 
> are package scoped.
>
> Unless the lucene jar is sealed, you should be able to add this class 
> to some other directory or jar in your classpath that has the correct 
> package structure without having to recompile the whole of Lucene.
>
> We saw a few of these go away in the move from Lucene 1.2 to 1.3 where 
> there were some visibilty changes to a few classes. IndexReader was 
> not one of them.
>
> I've been thinking a bit about my implementation, I don't think it 
> would be safe to call delete on this IndexReader since the document 
> number used when doing a search could change if the reader delegate is 
> closed and reopened between the search that got the document number 
> and the time delete is called (this is also the reason it could give 
> you inconsistent search results). It should probably throw a 
> NotSupportedException on calls to delete. It should probably also be 
> modified so it you can tell it to only refresh when it is idle 
> (although on a busy system that might make using it's refresh 
> functionality somewhat pointless).
>
> Eric
>
> -----Original Message-----
> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
> Sent: Thursday, May 22, 2003 4:04 AM
> To: Lucene Developers List
> Subject: Re: Problem with long run IndexSearcher
>
>
> Eric,
>
> I have start trying to integrate your class in our application. I have 
> tried to move the class out of the lucene package to be able to
> use it without needing to recompile the whole lucene package.
>
> I get this error:
> src/java/com/extrapola/search/
> IdleTimeoutRefreshingIndexReader.java:400: directory is not public in
> org.apache.lucene.index.IndexReader; cannot be accessed from outside
> package
>                                          synchronized (directory) { //
> in- & inter-process sync
>
> As far as I can understand it, this means that your code can not be
> inserted in an external package due to limited visibility granted to
> the directory attribute in the lucene code.
>
> Is this correct? Is there any reason to avaid this kind of extensions
> in other packages?
>
> Thanks for your attention,
>
> Giulio Cesare Solaroli
>
> On Monday, May 19, 2003, at 18:03 Europe/Rome, Eric Isakson wrote:
>
>> Hope it works like I planned it to :-) If you have any performance
>> metrics, I'd be interested to see how much it affects your search
>> performance.
>>
>> Eric
>>
>> -----Original Message-----
>> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
>> Sent: Monday, May 19, 2003 10:59 AM
>> To: Lucene Developers List
>> Subject: Re: Problem with long run IndexSearcher
>>
>>
>> Eric,
>>
>> On Monday, May 19, 2003, at 16:52 Europe/Rome, Eric Isakson wrote:
>>
>>> I was just reading Karsten's note, I'm not sure what impact my
>>> strategy will have if your application pages through hit results over
>>> a period of time...might not work at all (or perhaps inconsistently).
>>> The application I use recalculates search  results for each page as I
>>> page through hits rather than trying to hold onto the results from
>>> page to page. If the reader changed to an index that resulted in
>>> different hits it would seem inconsistent. You would only notice the
>>> inconsistency if you happened to be paging around while my reader was
>>> being refreshed :(
>>
>> At the moment we re-run the query to page through a large result set.
>> This could cause some inconsistency in the results, but the problem is
>> really marginal in our environment.
>>
>>> Hmph, guess I didn't really solve the problem with this new class,
>>> but it is a start on a solution and I'd be happy to keep working.
>>
>> For us it could be a definite solution, even if I think that a more
>> elegant solution could give Lucene a final boost in becoming the
>> ultimate solution for searching!! ;-]
>>
>> Giulio Cesare
>>
>>
>>> -----Original Message-----
>>> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
>>> Sent: Monday, May 19, 2003 10:44 AM
>>> To: Lucene Developers List
>>> Subject: Re: Problem with long run IndexSearcher
>>>
>>>
>>> Hi Eric,
>>>
>>> On Monday, May 19, 2003, at 16:18 Europe/Rome, Eric Isakson wrote:
>>>
>>>> bah, accidently sent that before I was finished typing...
>>>>
>>>> see:
>>>> http://nagoya.apache.org/eyebrowse/ReadMsg?listName=lucene-
>>>> user@jakarta.apache.org&msgNo=1859
>>>>
>>>> which really should be added to the FAQ.
>>>>
>>>> It just so happens that I spent a good deal of my weekend working on
>>>> a class to solve this problem. It hasn't been tested much and I'm
>>>> new to writing apps that deal with threading issues (so I don't know
>>>> if I blundered anywhere). I'd love to see if it works and get
>>>> feedback from anyone that uses it.
>>>
>>> Great!!!
>>>
>>>> See my message at:
>>>>
>>>> http://nagoya.apache.org/eyebrowse/ReadMsg?listName=lucene-
>>>> dev@jakarta.apache.org&msgNo=3393
>>>
>>> I did read it, but not carefully enught to realize it was what I was
>>> looking for. I will give it a try!!
>>>
>>>> If you have any feedback, pleass attach comments to the bug.
>>>
>>> Ok. I will let you know.
>>>
>>> Thanks again for your support.
>>>
>>> Giulio Cesare
>>>
>>>
>>>
>>>> Eric
>>>>
>>>> --
>>>> Eric D. Isakson        SAS Institute Inc.
>>>> Application Developer  SAS Campus Drive
>>>> XML Technologies       Cary, NC 27513
>>>> (919) 531-3639         http://www.sas.com
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Eric Isakson [mailto:Eric.Isakson@sas.com]
>>>> Sent: Monday, May 19, 2003 10:05 AM
>>>> To: Lucene Developers List
>>>> Subject: RE: Problem with long run IndexSearcher
>>>>
>>>>
>>>> Giulio,
>>>>
>>>> This problem stems from the fact that the IndexReader won't see your
>>>> changed index.
>>>>
>>>> See the faq entries:
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Giulio Cesare Solaroli [mailto:slrgcsa@ibn-italy.com]
>>>> Sent: Monday, May 19, 2003 9:38 AM
>>>> To: lucene-dev@jakarta.apache.org
>>>> Subject: Problem with long run IndexSearcher
>>>>
>>>>
>>>> Hi all,
>>>>
>>>> first let me express my compliments for Lucene.
>>>> I have been up for a full week-end to double check the results I was
>>>> having because I couldn't belive what I saw; with a stupid
>>>> application I could index DB data at a sustained rate of 50
>>>> documents per second.
>>>>
>>>> Now we have more that 2 millions documents indexed and the
>>>> performance are still excellent; our main bottle neck is still the
>>>> DB.
>>>>
>>>> Our situation:
>>>> - we are indexing new documents at a sustained rate (an average of
>>>> 40.000 new documents a day);
>>>> - we have written a small xmlRpc server in Java to search the index
>>>> from other applications.
>>>>
>>>> The xmlRpc server creates a single instance of IndexSearcher a reuse
>>>> it for each query issued. For each request, a new Query object is
>>>> created and the documents found are returned to the client.
>>>>
>>>> The problem we are seeing is that the documents indexed after the
>>>> xmlRpc server is started will not be found until the server is
>>>> restarted.
>>>>
>>>> Is this our foult, or the way IndexSearcher should work?
>>>>
>>>> What is the best way to keep the IndexSearcher up to date with the
>>>> updated index?
>>>>
>>>> Thanks for your attention,
>>>>
>>>> Giulio Cesare Solaroli
>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> -
>>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message