lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Index Partitioning ( was Re: Search deadlocking under load)
Date Mon, 11 Jul 2005 20:11:46 GMT
If you want really real-time updates of search results, then yes. 
However, maybe you can live with near-real-time results, in which cases
you can add some logic to your application to check for index version
only every N requests/minutes/hours.

Otis


--- Aalap Parikh <aloo77@yahoo.com> wrote:

> >I don't really know a lot about what gets loaded into
> memory when you
> >make/use a new searcher, but the one thing i've
> learned from experience 
> >is
> >that the FieldCache (which gets used when you sort on
> a field) contains
> >every term in the field you are sorting on, and an
> instance of 
> >FieldCache
> >exists for each IndexReader you open (which is one
> big reason not to 
> >open
> >a seperate reader for every client).
> 
> You mentioned that re-using the same IndexSearcher
> would provide better performance in terms of sorting
> of search results, but what if the index I am
> searching on is constantly being updated? Would using
> the same Searcher pick up those updates (adds/removes)
> or that I would need to instantiate a new Searcher for
> each of the client requests in order for the search
> results to reflect the updated or new docs in the
> index?
> 
> Thanks,
> Aalap.
> 
> --- Chris Hostetter <hossman_lucene@fucit.org> wrote:
> 
> > 
> > : > Generally speaking, you only ever need one
> > active Searcher, which
> > : > all of
> > : > your threads should be able to use.  (Of course,
> > Nathan says that
> > : > in his
> > : > code base, doing this causes his JVM to freeze
> > up, but I've never seen
> > : > this myself).
> > : >
> > : Thanks for your response Chris.  Do you think we
> > are going down a
> > : deadly path by having "many smaller"
> > IndexSearchers open rather than
> > : "one very large one"?
> > 
> > I'm sorry ... i think i may have confused you, i
> > forgot  that this thread
> > was regarding partioning the index.  i ment one
> > searcher *per index* ...
> > don't try to make a seperate searcher per client, or
> > have a pool of
> > searchers, or anything like that.  But if you have a
> > need to partition
> > your data into multiple indexes, then have one
> > searcher per index.
> > 
> > I don't really know a lot about what gets loaded
> > into memory when you
> > make/use a new searcher, but the one thing i've
> > learned from experience is
> > that the FieldCache (which gets used when you sort
> > on a field) contains
> > every term in the field you are sorting on, and an
> > instance of FieldCache
> > exists for each IndexReader you open (which is one
> > big reason not to open
> > a seperate reader for every client).
> > 
> > now assume you partition your data into two seperate
> > indexes, unless the
> > way you partition your data lets you cleanly so that
> > each of hte
> > two indexes contains only half the number of terms
> > as if you had one big
> > index, then sorting on a field in those two indexes
> > will require more RAM
> > then sorting on the same data in asingle index.
> > 
> > ...that's just one example i ran into when looking
> > at partitioning data,
> > i'm sure there are other cases where splitting your
> > data up into seperate
> > indexes isn't neccessarily more efficient then using
> > one big index.
> > 
> > 
> > 
> > -Hoss
> > 
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > 
> > 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around 
> http://mail.yahoo.com 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message