lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Woodward <a...@flax.co.uk>
Subject Re: Memory problem with TermQuery
Date Mon, 08 Jun 2015 10:55:51 GMT
You'll still need to call rewrite, but it needs to be done per-reader, so you'll need to cache
the queries *before* they're rewritten, and then call rewrite whenever you create a new IndexReader.
 Otherwise you'll get incorrect scores, and possibly missed hits as well.

Alan Woodward
www.flax.co.uk


On 8 Jun 2015, at 11:46, Anna Maier wrote:

> Hi Alan,
> 
> you are right, we are calling rewrite on our query at some point. Ok, it would probably
be an option to take that out.
> Thanks for the hint!
> 
> Best,
> Anna
> 
> -----Original Message-----
> From: Alan Woodward [mailto:alan@flax.co.uk] 
> Sent: Montag, 8. Juni 2015 12:23
> To: java-user@lucene.apache.org
> Subject: Re: Memory problem with TermQuery
> 
> Hi Anna,
> 
> In normal usage, perReaderTermState will be null, and TermQuery will be very lightweight.
 It's in particular expert use cases (generally after queries have been rewritten against
a specific IndexReader) that the perReaderTermState will be initialized.  Are you cacheing
rewritten queries somehow?
> 
> Alan Woodward
> www.flax.co.uk
> 
> 
> On 8 Jun 2015, at 10:49, Anna Maier wrote:
> 
>> Hi,
>> 
>> we ran into a memory problem with TermQuery: in our program, we build a TermQuery
object from the user input and pass it around, to be able to different things, like execute
the query again and so on. So, the TermQuery object can potentially exist for some time.
>> Now it turns out, that a TermQuery keeps a reference to an IndexReader (via the perReaderTermState
field). 
>> This keeps our program from throwing old readers away when new ones are opened. This
has quite an impact on the required memory, especially for big indices. It is not feasible
anymore now to keep a reference to a TermQuery for longer.
>> 
>> I'm wondering: is this a bug? After all, I would have expected the TermQuery to be
a lightweight object. Or is the TermQuery not intended to be passed around in the program
at all? 
>> 
>> Best,
>> Anna
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message